Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250629 01:51:35.872923 416 test_util.cc:276] Using random seed: 650078589
2025-06-29T01:51:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-29T01:51:35Z Disabled control of system clock
I20250629 01:51:35.901376 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34747
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250629 01:51:36.167021 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:36.167256 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:36.167309 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250629 01:51:36.173830 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250629 01:51:36.173893 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:36.173936 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250629 01:51:36.173967 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250629 01:51:36.181245 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:36.182390 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:36.183796 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:36.192592 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:36.192674 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:36.192628 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:36.193117 424 server_base.cc:1048] running on GCE node
I20250629 01:51:36.193709 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:36.194753 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:36.195931 424 hybrid_clock.cc:648] HybridClock initialized: now 1751161896195866 us; error 57 us; skew 500 ppm
I20250629 01:51:36.196241 424 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:36.205314 424 webserver.cc:469] Webserver started at http://127.0.104.62:36579/ using document root <none> and password file <none>
I20250629 01:51:36.205960 424 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:36.206053 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:36.206349 424 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250629 01:51:36.208410 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data/instance:
uuid: "a0b76160e22c46929936cb9682322fae"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.209038 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal/instance:
uuid: "a0b76160e22c46929936cb9682322fae"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.213243 424 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20250629 01:51:36.216024 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:36.217360 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250629 01:51:36.217515 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal
uuid: "a0b76160e22c46929936cb9682322fae"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.217706 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:36.233909 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:36.234488 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:36.234666 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:36.252185 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34747
I20250629 01:51:36.252241 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34747 every 8 connection(s)
I20250629 01:51:36.253307 424 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/data/info.pb
I20250629 01:51:36.256297 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:36.260855 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } has no permanent_uuid. Determining permanent_uuid...
I20250629 01:51:36.263036 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 424
I20250629 01:51:36.263279 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-0/wal/instance
I20250629 01:51:36.267102 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35873
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250629 01:51:36.277005 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
W20250629 01:51:36.277843 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111)
W20250629 01:51:36.279080 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111)
I20250629 01:51:36.328424 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } attempt: 1
W20250629 01:51:36.330013 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111)
I20250629 01:51:36.398262 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } attempt: 2
W20250629 01:51:36.399895 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111)
I20250629 01:51:36.491258 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } attempt: 3
W20250629 01:51:36.492913 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:35873: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111)
W20250629 01:51:36.541007 494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:36.541335 494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:36.541496 494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250629 01:51:36.548130 494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250629 01:51:36.548197 494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:36.548241 494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250629 01:51:36.548280 494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250629 01:51:36.555936 494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:36.557144 494 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:36.558542 494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:36.567044 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:36.567032 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:36.567073 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:36.568533 494 server_base.cc:1048] running on GCE node
I20250629 01:51:36.569118 494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:36.570212 494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:36.571421 494 hybrid_clock.cc:648] HybridClock initialized: now 1751161896571381 us; error 33 us; skew 500 ppm
I20250629 01:51:36.571676 494 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:36.573511 494 webserver.cc:469] Webserver started at http://127.0.104.61:43231/ using document root <none> and password file <none>
I20250629 01:51:36.574079 494 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:36.574167 494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:36.574442 494 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250629 01:51:36.576319 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/instance:
uuid: "0f4258c004e04bcf8216bb7c8fd56c54"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.576920 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal/instance:
uuid: "0f4258c004e04bcf8216bb7c8fd56c54"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.580928 494 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250629 01:51:36.583561 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:36.584592 494 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250629 01:51:36.584713 494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
uuid: "0f4258c004e04bcf8216bb7c8fd56c54"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.584863 494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:36.608987 494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:36.609570 494 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:36.609766 494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:36.627435 494 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:35873
I20250629 01:51:36.627460 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:35873 every 8 connection(s)
I20250629 01:51:36.628540 494 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
I20250629 01:51:36.631474 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:36.633329 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 494
I20250629 01:51:36.633450 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal/instance
I20250629 01:51:36.634408 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:32789
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250629 01:51:36.636152 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } attempt: 4
I20250629 01:51:36.636536 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } has no permanent_uuid. Determining permanent_uuid...
I20250629 01:51:36.648269 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
I20250629 01:51:36.649091 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } has no permanent_uuid. Determining permanent_uuid...
W20250629 01:51:36.650868 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.653376 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } has no permanent_uuid. Determining permanent_uuid...
W20250629 01:51:36.654119 511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
W20250629 01:51:36.655678 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.704901 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 1
W20250629 01:51:36.706369 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.710070 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 1
W20250629 01:51:36.711472 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.774561 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 2
W20250629 01:51:36.776167 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.778653 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 2
W20250629 01:51:36.779945 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.867450 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 3
W20250629 01:51:36.869222 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:36.880175 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 3
W20250629 01:51:36.881552 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
W20250629 01:51:36.921135 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:36.921360 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:36.921411 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250629 01:51:36.927506 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250629 01:51:36.927567 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:36.927608 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250629 01:51:36.927639 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250629 01:51:36.935273 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:36.936471 563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:36.937956 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:36.946832 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:36.946916 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:36.946884 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:36.947872 563 server_base.cc:1048] running on GCE node
I20250629 01:51:36.948462 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:36.949486 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:36.950666 563 hybrid_clock.cc:648] HybridClock initialized: now 1751161896950633 us; error 26 us; skew 500 ppm
I20250629 01:51:36.950932 563 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:36.952702 563 webserver.cc:469] Webserver started at http://127.0.104.60:34121/ using document root <none> and password file <none>
I20250629 01:51:36.953459 563 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:36.953567 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:36.953863 563 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250629 01:51:36.955775 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/instance:
uuid: "f304077528b34c61857a659e5873631d"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.956439 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal/instance:
uuid: "f304077528b34c61857a659e5873631d"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.961175 563 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250629 01:51:36.963723 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:36.964777 563 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250629 01:51:36.964911 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
uuid: "f304077528b34c61857a659e5873631d"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:36.965059 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:37.003756 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:37.004374 563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:37.004568 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:37.012488 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 4
W20250629 01:51:37.014217 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111)
I20250629 01:51:37.021618 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:32789
I20250629 01:51:37.021647 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:32789 every 8 connection(s)
I20250629 01:51:37.023142 563 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/info.pb
I20250629 01:51:37.026157 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:37.028468 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 563
I20250629 01:51:37.028621 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal/instance
I20250629 01:51:37.030810 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } has no permanent_uuid. Determining permanent_uuid...
I20250629 01:51:37.051750 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 4
I20250629 01:51:37.054692 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } has no permanent_uuid. Determining permanent_uuid...
I20250629 01:51:37.061498 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } has no permanent_uuid. Determining permanent_uuid...
I20250629 01:51:37.064046 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Bootstrap starting.
I20250629 01:51:37.067157 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:37.068001 490 log.cc:826] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:37.069056 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Bootstrap starting.
I20250629 01:51:37.069988 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: No bootstrap required, opened a new log
I20250629 01:51:37.071228 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:37.072301 632 log.cc:826] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:37.074074 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: No bootstrap required, opened a new log
I20250629 01:51:37.074620 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.074833 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:37.074913 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b76160e22c46929936cb9682322fae, State: Initialized, Role: FOLLOWER
I20250629 01:51:37.075397 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.076079 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.076357 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.076777 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: configured and running, proceeding with master startup.
I20250629 01:51:37.078702 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.078912 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:37.078989 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f304077528b34c61857a659e5873631d, State: Initialized, Role: FOLLOWER
I20250629 01:51:37.079391 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.080703 643 sys_catalog.cc:455] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.081022 643 sys_catalog.cc:458] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.081137 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: configured and running, proceeding with master startup.
I20250629 01:51:37.087488 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:34747: OK, instance_id { permanent_uuid: "a0b76160e22c46929936cb9682322fae" instance_seqno: 1751161896247596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250629 01:51:37.094352 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250629 01:51:37.096570 657 catalog_manager.cc:1560] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250629 01:51:37.096652 657 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250629 01:51:37.098794 665 catalog_manager.cc:1560] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250629 01:51:37.098901 665 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250629 01:51:37.097532 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:35873: OK, instance_id { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" instance_seqno: 1751161896622490 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250629 01:51:37.103192 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250629 01:51:37.104710 641 raft_consensus.cc:491] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:37.104893 641 raft_consensus.cc:513] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.106251 641 leader_election.cc:290] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873), f304077528b34c61857a659e5873631d (127.0.104.60:32789)
I20250629 01:51:37.107544 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0b76160e22c46929936cb9682322fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f304077528b34c61857a659e5873631d" is_pre_election: true
I20250629 01:51:37.107939 607 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b76160e22c46929936cb9682322fae in term 0.
I20250629 01:51:37.108469 440 leader_election.cc:304] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b76160e22c46929936cb9682322fae, f304077528b34c61857a659e5873631d; no voters:
I20250629 01:51:37.108846 641 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:37.108974 641 raft_consensus.cc:491] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:37.109076 641 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:37.112346 641 raft_consensus.cc:513] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.113123 641 leader_election.cc:290] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [CANDIDATE]: Term 1 election: Requested vote from peers 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873), f304077528b34c61857a659e5873631d (127.0.104.60:32789)
I20250629 01:51:37.113917 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0b76160e22c46929936cb9682322fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f304077528b34c61857a659e5873631d"
I20250629 01:51:37.114073 607 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:37.116878 607 raft_consensus.cc:2466] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b76160e22c46929936cb9682322fae in term 1.
I20250629 01:51:37.117296 440 leader_election.cc:304] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b76160e22c46929936cb9682322fae, f304077528b34c61857a659e5873631d; no voters:
I20250629 01:51:37.117659 641 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:37.118304 641 raft_consensus.cc:695] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [term 1 LEADER]: Becoming Leader. State: Replica: a0b76160e22c46929936cb9682322fae, State: Running, Role: LEADER
I20250629 01:51:37.119040 641 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.123930 667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b76160e22c46929936cb9682322fae. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.124264 667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:51:37.136713 669 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250629 01:51:37.138507 669 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250629 01:51:37.147776 607 raft_consensus.cc:1273] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Refusing update from remote peer a0b76160e22c46929936cb9682322fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:37.148572 667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:37.173732 643 sys_catalog.cc:455] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b76160e22c46929936cb9682322fae. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.173919 643 sys_catalog.cc:458] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.174280 670 mvcc.cc:204] Tried to move back new op lower bound from 7172759130706407424 to 7172759130608250880. Current Snapshot: MvccSnapshot[applied={T|T < 7172759130706407424}]
I20250629 01:51:37.185035 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.185226 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:51:37.186852 669 catalog_manager.cc:1349] Generated new cluster ID: 3bd8135387974b63b91ae240fae34427
I20250629 01:51:37.186937 669 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250629 01:51:37.186975 643 sys_catalog.cc:455] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.187124 643 sys_catalog.cc:458] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.209628 669 catalog_manager.cc:1372] Generated new certificate authority record
I20250629 01:51:37.211862 669 catalog_manager.cc:1506] Loading token signing keys...
I20250629 01:51:37.228610 669 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Generated new TSK 0
I20250629 01:51:37.229740 669 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250629 01:51:37.248251 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250629 01:51:37.313406 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } attempt: 5
I20250629 01:51:37.326121 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Bootstrap starting.
I20250629 01:51:37.332154 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:37.336220 561 log.cc:826] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:37.346500 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: No bootstrap required, opened a new log
I20250629 01:51:37.357321 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.357882 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:37.358127 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4258c004e04bcf8216bb7c8fd56c54, State: Initialized, Role: FOLLOWER
I20250629 01:51:37.359181 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.371299 534 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:37.372169 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0b76160e22c46929936cb9682322fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" is_pre_election: true
I20250629 01:51:37.372740 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a0b76160e22c46929936cb9682322fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4258c004e04bcf8216bb7c8fd56c54"
I20250629 01:51:37.376056 534 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Refusing update from remote peer a0b76160e22c46929936cb9682322fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:37.376811 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: configured and running, proceeding with master startup.
I20250629 01:51:37.376818 667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:37.443002 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.443045 679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b76160e22c46929936cb9682322fae. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.443064 686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.443449 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.443451 679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.443450 686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:37.473932 694 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250629 01:51:37.678787 675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:37.679021 675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:37.679092 675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250629 01:51:37.685214 675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:37.685334 675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250629 01:51:37.693065 675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:37.694181 675 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:37.695561 675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:37.705504 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:37.705440 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:37.705479 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:37.705825 675 server_base.cc:1048] running on GCE node
I20250629 01:51:37.706394 675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:37.707129 675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:37.708321 675 hybrid_clock.cc:648] HybridClock initialized: now 1751161897708296 us; error 27 us; skew 500 ppm
I20250629 01:51:37.708560 675 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:37.710333 675 webserver.cc:469] Webserver started at http://127.0.104.1:43533/ using document root <none> and password file <none>
I20250629 01:51:37.710942 675 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:37.711035 675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:37.711318 675 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250629 01:51:37.713233 675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data/instance:
uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
format_stamp: "Formatted at 2025-06-29 01:51:37 on dist-test-slave-0pfc"
I20250629 01:51:37.713852 675 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/instance:
uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
format_stamp: "Formatted at 2025-06-29 01:51:37 on dist-test-slave-0pfc"
I20250629 01:51:37.717840 675 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250629 01:51:37.720788 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:37.721748 675 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250629 01:51:37.721868 675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal
uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
format_stamp: "Formatted at 2025-06-29 01:51:37 on dist-test-slave-0pfc"
I20250629 01:51:37.722029 675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:37.729648 713 raft_consensus.cc:491] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0b76160e22c46929936cb9682322fae)
I20250629 01:51:37.729830 713 raft_consensus.cc:513] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.730809 713 leader_election.cc:290] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0b76160e22c46929936cb9682322fae (127.0.104.62:34747), 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873)
I20250629 01:51:37.731357 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f304077528b34c61857a659e5873631d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" is_pre_election: true
I20250629 01:51:37.731403 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f304077528b34c61857a659e5873631d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0b76160e22c46929936cb9682322fae" is_pre_election: true
I20250629 01:51:37.732066 582 leader_election.cc:304] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f304077528b34c61857a659e5873631d; no voters: 0f4258c004e04bcf8216bb7c8fd56c54, a0b76160e22c46929936cb9682322fae
I20250629 01:51:37.732379 713 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:37.736842 675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:37.737401 675 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:37.737593 675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:37.738291 675 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250629 01:51:37.739928 675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250629 01:51:37.740010 675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:37.740126 675 ts_tablet_manager.cc:610] Registered 0 tablets
I20250629 01:51:37.740193 675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250629 01:51:37.771606 675 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37891
I20250629 01:51:37.771675 821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37891 every 8 connection(s)
I20250629 01:51:37.773032 675 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/data/info.pb
I20250629 01:51:37.782150 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 675
I20250629 01:51:37.782341 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/instance
I20250629 01:51:37.786746 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250629 01:51:37.790138 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:37.790416 822 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:37.790629 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:37.790751 824 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:37.791348 822 heartbeater.cc:507] Master 127.0.104.60:32789 requested a full tablet report, sending...
I20250629 01:51:37.791365 824 heartbeater.cc:507] Master 127.0.104.61:35873 requested a full tablet report, sending...
I20250629 01:51:37.793591 597 ts_manager.cc:194] Registered new tserver with Master: ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:37.794879 526 ts_manager.cc:194] Registered new tserver with Master: ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:37.795038 823 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34747
I20250629 01:51:37.795145 823 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:37.795487 823 heartbeater.cc:507] Master 127.0.104.62:34747 requested a full tablet report, sending...
I20250629 01:51:37.796703 454 ts_manager.cc:194] Registered new tserver with Master: ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:37.798071 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:33711
I20250629 01:51:37.889124 686 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0b76160e22c46929936cb9682322fae)
I20250629 01:51:37.889364 686 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:37.890537 686 leader_election.cc:290] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0b76160e22c46929936cb9682322fae (127.0.104.62:34747), f304077528b34c61857a659e5873631d (127.0.104.60:32789)
I20250629 01:51:37.890990 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0b76160e22c46929936cb9682322fae" is_pre_election: true
I20250629 01:51:37.891010 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f304077528b34c61857a659e5873631d" is_pre_election: true
I20250629 01:51:37.891566 511 leader_election.cc:304] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f4258c004e04bcf8216bb7c8fd56c54; no voters: a0b76160e22c46929936cb9682322fae, f304077528b34c61857a659e5873631d
I20250629 01:51:37.891842 686 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:37.964380 667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:37.964623 667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
W20250629 01:51:38.067263 831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:38.067479 831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:38.067592 831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250629 01:51:38.074185 831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:38.074330 831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250629 01:51:38.081566 831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:38.082672 831 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:38.084041 831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:38.093950 840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:38.093977 839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:38.094074 842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:38.094339 831 server_base.cc:1048] running on GCE node
I20250629 01:51:38.094962 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:38.095664 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:38.096837 831 hybrid_clock.cc:648] HybridClock initialized: now 1751161898096795 us; error 35 us; skew 500 ppm
I20250629 01:51:38.097079 831 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:38.098083 657 catalog_manager.cc:1261] Loaded cluster ID: 3bd8135387974b63b91ae240fae34427
I20250629 01:51:38.098150 657 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: loading cluster ID for follower catalog manager: success
I20250629 01:51:38.099413 831 webserver.cc:469] Webserver started at http://127.0.104.2:33171/ using document root <none> and password file <none>
I20250629 01:51:38.099978 831 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:38.100057 831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:38.100342 831 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250629 01:51:38.100488 657 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: acquiring CA information for follower catalog manager: success
I20250629 01:51:38.101920 657 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250629 01:51:38.102295 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data/instance:
uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
format_stamp: "Formatted at 2025-06-29 01:51:38 on dist-test-slave-0pfc"
I20250629 01:51:38.102918 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/instance:
uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
format_stamp: "Formatted at 2025-06-29 01:51:38 on dist-test-slave-0pfc"
I20250629 01:51:38.107007 831 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250629 01:51:38.109684 848 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:38.110659 831 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:38.110772 831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal
uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
format_stamp: "Formatted at 2025-06-29 01:51:38 on dist-test-slave-0pfc"
I20250629 01:51:38.110924 831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:38.146637 831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:38.147244 831 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:38.147440 831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:38.148206 831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250629 01:51:38.149421 831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250629 01:51:38.149493 831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:38.149586 831 ts_tablet_manager.cc:610] Registered 0 tablets
I20250629 01:51:38.149639 831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:38.178984 831 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33393
I20250629 01:51:38.179046 960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33393 every 8 connection(s)
I20250629 01:51:38.179951 831 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/data/info.pb
I20250629 01:51:38.182621 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 831
I20250629 01:51:38.182798 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/instance
I20250629 01:51:38.189463 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250629 01:51:38.196676 961 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:38.197000 961 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:38.197840 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:38.197957 961 heartbeater.cc:507] Master 127.0.104.60:32789 requested a full tablet report, sending...
I20250629 01:51:38.197975 963 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:38.198585 963 heartbeater.cc:507] Master 127.0.104.61:35873 requested a full tablet report, sending...
I20250629 01:51:38.199491 526 ts_manager.cc:194] Registered new tserver with Master: 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:38.199502 597 ts_manager.cc:194] Registered new tserver with Master: 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:38.200914 962 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34747
I20250629 01:51:38.201007 962 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:38.201359 962 heartbeater.cc:507] Master 127.0.104.62:34747 requested a full tablet report, sending...
I20250629 01:51:38.202119 454 ts_manager.cc:194] Registered new tserver with Master: 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:38.202844 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:41237
I20250629 01:51:38.272042 969 raft_consensus.cc:491] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0b76160e22c46929936cb9682322fae)
I20250629 01:51:38.272225 969 raft_consensus.cc:513] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:38.273201 969 leader_election.cc:290] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0b76160e22c46929936cb9682322fae (127.0.104.62:34747), 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873)
I20250629 01:51:38.273677 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f304077528b34c61857a659e5873631d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a0b76160e22c46929936cb9682322fae" is_pre_election: true
I20250629 01:51:38.273671 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f304077528b34c61857a659e5873631d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" is_pre_election: true
I20250629 01:51:38.274356 584 leader_election.cc:304] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f304077528b34c61857a659e5873631d; no voters: 0f4258c004e04bcf8216bb7c8fd56c54, a0b76160e22c46929936cb9682322fae
I20250629 01:51:38.274642 969 raft_consensus.cc:2747] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:38.299841 823 heartbeater.cc:499] Master 127.0.104.62:34747 was elected leader, sending a full tablet report...
W20250629 01:51:38.475972 968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:38.476253 968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:38.476379 968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250629 01:51:38.478091 694 catalog_manager.cc:1261] Loaded cluster ID: 3bd8135387974b63b91ae240fae34427
I20250629 01:51:38.478191 694 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: loading cluster ID for follower catalog manager: success
I20250629 01:51:38.481395 694 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: acquiring CA information for follower catalog manager: success
W20250629 01:51:38.483256 968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250629 01:51:38.483371 694 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250629 01:51:38.483418 968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250629 01:51:38.491240 968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:38.492380 968 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:38.493722 968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:38.504216 978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:38.504268 976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:38.504216 975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:38.505125 968 server_base.cc:1048] running on GCE node
I20250629 01:51:38.505591 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:38.506351 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:38.507545 968 hybrid_clock.cc:648] HybridClock initialized: now 1751161898507499 us; error 42 us; skew 500 ppm
I20250629 01:51:38.507771 968 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:38.509783 968 webserver.cc:469] Webserver started at http://127.0.104.3:34579/ using document root <none> and password file <none>
I20250629 01:51:38.510339 968 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:38.510408 968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:38.510655 968 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250629 01:51:38.512511 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data/instance:
uuid: "4712643b602e4f898752fcab6a1e5913"
format_stamp: "Formatted at 2025-06-29 01:51:38 on dist-test-slave-0pfc"
I20250629 01:51:38.513134 968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/instance:
uuid: "4712643b602e4f898752fcab6a1e5913"
format_stamp: "Formatted at 2025-06-29 01:51:38 on dist-test-slave-0pfc"
I20250629 01:51:38.517777 968 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250629 01:51:38.521093 984 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:38.522145 968 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250629 01:51:38.522269 968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal
uuid: "4712643b602e4f898752fcab6a1e5913"
format_stamp: "Formatted at 2025-06-29 01:51:38 on dist-test-slave-0pfc"
I20250629 01:51:38.522418 968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:38.534920 968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:38.535506 968 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:38.535691 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:38.536505 968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250629 01:51:38.537742 968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250629 01:51:38.537823 968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:38.537914 968 ts_tablet_manager.cc:610] Registered 0 tablets
I20250629 01:51:38.537977 968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:38.571067 968 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:33889
I20250629 01:51:38.571112 1096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:33889 every 8 connection(s)
I20250629 01:51:38.572067 968 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/data/info.pb
I20250629 01:51:38.577416 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 968
I20250629 01:51:38.577570 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/instance
I20250629 01:51:38.596308 1098 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34747
I20250629 01:51:38.596596 1098 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:38.596873 1097 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:38.596985 1097 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:38.597514 1098 heartbeater.cc:507] Master 127.0.104.62:34747 requested a full tablet report, sending...
I20250629 01:51:38.597566 1097 heartbeater.cc:507] Master 127.0.104.60:32789 requested a full tablet report, sending...
I20250629 01:51:38.598958 454 ts_manager.cc:194] Registered new tserver with Master: 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:38.599016 597 ts_manager.cc:194] Registered new tserver with Master: 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:38.599817 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:33517
I20250629 01:51:38.604713 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:38.604806 1099 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:38.605033 1099 heartbeater.cc:507] Master 127.0.104.61:35873 requested a full tablet report, sending...
I20250629 01:51:38.605783 526 ts_manager.cc:194] Registered new tserver with Master: 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:38.614929 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250629 01:51:38.647697 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6cda0cad0c0341b0ba9f5320b745beab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.647714 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4e5ceee7b9ba4c989df3896749afaa74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.647665 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9c3c84a9cd824ea195a4c0768fd1d42b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.649144 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_541785013ee0461c8f133bf66165e090"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.649477 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b4c3372a7dda48ac8963c3372b851346"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.648320 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dc24e0b237234c08b6d456711117cf60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.650477 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a8a745e0c9cd4b05a8a3300ccacc12a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.650955 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8a745e0c9cd4b05a8a3300ccacc12a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.651134 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c3c84a9cd824ea195a4c0768fd1d42b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.651172 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc24e0b237234c08b6d456711117cf60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.650918 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e5ceee7b9ba4c989df3896749afaa74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.651619 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4c3372a7dda48ac8963c3372b851346 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.652019 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_541785013ee0461c8f133bf66165e090 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.650920 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cda0cad0c0341b0ba9f5320b745beab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.657389 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5da38d95bc4b4788bdfd6ebcdafa4621"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.657842 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5da38d95bc4b4788bdfd6ebcdafa4621 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.660569 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_963d2af04f874c58a54abb3ac76416a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.661154 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_963d2af04f874c58a54abb3ac76416a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.668794 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7930964e266c461ba221f846be3d7ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.669417 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7930964e266c461ba221f846be3d7ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.693131 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6cda0cad0c0341b0ba9f5320b745beab" } new_table_name: "default.Table_7017fd5ca5754ea0b74a91ffa9017025" modify_external_catalogs: true
I20250629 01:51:38.704484 962 heartbeater.cc:499] Master 127.0.104.62:34747 was elected leader, sending a full tablet report...
I20250629 01:51:38.732796 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9c3c84a9cd824ea195a4c0768fd1d42b" } new_table_name: "default.Table_bf8dc25dca7744e5b2ba3b14f1713b89" modify_external_catalogs: true
I20250629 01:51:38.733101 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5da38d95bc4b4788bdfd6ebcdafa4621" } modify_external_catalogs: true
I20250629 01:51:38.733314 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5da38d95bc4b4788bdfd6ebcdafa4621" } modify_external_catalogs: true
I20250629 01:51:38.734231 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dc24e0b237234c08b6d456711117cf60" } modify_external_catalogs: true
I20250629 01:51:38.734453 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dc24e0b237234c08b6d456711117cf60" } modify_external_catalogs: true
I20250629 01:51:38.739919 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b4c3372a7dda48ac8963c3372b851346" } new_table_name: "default.Table_4fac489880854041a135f94794583e93" modify_external_catalogs: true
I20250629 01:51:38.741250 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a8a745e0c9cd4b05a8a3300ccacc12a9" } new_table_name: "default.Table_55740e7f990f470d86e7e7fc429b6d79" modify_external_catalogs: true
I20250629 01:51:38.742517 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4e5ceee7b9ba4c989df3896749afaa74" } new_table_name: "default.Table_ca6aba8125c9496d86aa4dff2eed2591" modify_external_catalogs: true
I20250629 01:51:38.743739 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_541785013ee0461c8f133bf66165e090" } modify_external_catalogs: true
I20250629 01:51:38.743932 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_541785013ee0461c8f133bf66165e090" } modify_external_catalogs: true
W20250629 01:51:38.775283 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da27c51828684053b4f777739b0f2e51 consensus state has no leader
W20250629 01:51:38.776830 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8453efdd649747a78c94eaa82fcde378 has no consensus state
I20250629 01:51:38.778409 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet cf53ffe352b948ecac63fb19761b9b4f
I20250629 01:51:38.781538 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7930964e266c461ba221f846be3d7ea5" } modify_external_catalogs: true
I20250629 01:51:38.781718 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7930964e266c461ba221f846be3d7ea5" } modify_external_catalogs: true
I20250629 01:51:38.787245 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 58b408302cc448eba34e0c0c67b25443
W20250629 01:51:38.795075 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17dfc03d30ed40a498cae5efdc5445c7 has no consensus state
W20250629 01:51:38.795482 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9af16f666fc6479ab67e449d1e962f8f has no consensus state
W20250629 01:51:38.795815 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b966a9c2749640808cf97059845d3b44 has no consensus state
I20250629 01:51:38.811223 448 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c42e3b8bc18e43e5b0c8b68dec0ca407
I20250629 01:51:38.811535 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a8eb0fa19ae84df79346736446193b4b
W20250629 01:51:38.817028 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 8453efdd649747a78c94eaa82fcde378 (table default.Table_bf8dc25dca7744e5b2ba3b14f1713b89 [id=00acceb7e9af4dc8b6fe843e436bf156], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8453efdd649747a78c94eaa82fcde378 has no consensus state
I20250629 01:51:38.831792 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b966a9c2749640808cf97059845d3b44 (DEFAULT_TABLE table=default.Table_ca6aba8125c9496d86aa4dff2eed2591 [id=d4e47fc75da249138358461212d80e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.832309 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca6aba8125c9496d86aa4dff2eed2591" } modify_external_catalogs: true
I20250629 01:51:38.832450 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca6aba8125c9496d86aa4dff2eed2591" } modify_external_catalogs: true
I20250629 01:51:38.832926 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b966a9c2749640808cf97059845d3b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.832767 755 tablet_service.cc:1468] Processing CreateTablet for tablet b966a9c2749640808cf97059845d3b44 (DEFAULT_TABLE table=default.Table_ca6aba8125c9496d86aa4dff2eed2591 [id=d4e47fc75da249138358461212d80e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.833961 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b966a9c2749640808cf97059845d3b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.834638 752 tablet_service.cc:1468] Processing CreateTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 (DEFAULT_TABLE table=default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.835070 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17dfc03d30ed40a498cae5efdc5445c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.832815 754 tablet_service.cc:1468] Processing CreateTablet for tablet 9af16f666fc6479ab67e449d1e962f8f (DEFAULT_TABLE table=default.Table_55740e7f990f470d86e7e7fc429b6d79 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.835496 757 tablet_service.cc:1468] Processing CreateTablet for tablet da27c51828684053b4f777739b0f2e51 (DEFAULT_TABLE table=default.Table_6cda0cad0c0341b0ba9f5320b745beab [id=f4a6794121364ee19c24ef8ff99d8f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.835973 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af16f666fc6479ab67e449d1e962f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.836495 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da27c51828684053b4f777739b0f2e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.837116 753 tablet_service.cc:1468] Processing CreateTablet for tablet 8453efdd649747a78c94eaa82fcde378 (DEFAULT_TABLE table=default.Table_bf8dc25dca7744e5b2ba3b14f1713b89 [id=00acceb7e9af4dc8b6fe843e436bf156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.838634 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 8453efdd649747a78c94eaa82fcde378 (DEFAULT_TABLE table=default.Table_bf8dc25dca7744e5b2ba3b14f1713b89 [id=00acceb7e9af4dc8b6fe843e436bf156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.839071 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8453efdd649747a78c94eaa82fcde378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.840027 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 (DEFAULT_TABLE table=default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.840468 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17dfc03d30ed40a498cae5efdc5445c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.841171 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 9af16f666fc6479ab67e449d1e962f8f (DEFAULT_TABLE table=default.Table_55740e7f990f470d86e7e7fc429b6d79 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.841549 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af16f666fc6479ab67e449d1e962f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.842343 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8453efdd649747a78c94eaa82fcde378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.842620 1031 tablet_service.cc:1468] Processing CreateTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca (DEFAULT_TABLE table=default.Table_963d2af04f874c58a54abb3ac76416a6 [id=c22f2e9e8d084f2089def95c26c2b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.843015 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6690e00dfaa4d12be27e2fed9d835ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.843189 756 tablet_service.cc:1468] Processing CreateTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca (DEFAULT_TABLE table=default.Table_963d2af04f874c58a54abb3ac76416a6 [id=c22f2e9e8d084f2089def95c26c2b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.843549 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6690e00dfaa4d12be27e2fed9d835ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.830245 1032 tablet_service.cc:1468] Processing CreateTablet for tablet da27c51828684053b4f777739b0f2e51 (DEFAULT_TABLE table=default.Table_6cda0cad0c0341b0ba9f5320b745beab [id=f4a6794121364ee19c24ef8ff99d8f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.843834 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da27c51828684053b4f777739b0f2e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.844789 896 tablet_service.cc:1468] Processing CreateTablet for tablet da27c51828684053b4f777739b0f2e51 (DEFAULT_TABLE table=default.Table_6cda0cad0c0341b0ba9f5320b745beab [id=f4a6794121364ee19c24ef8ff99d8f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.845916 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da27c51828684053b4f777739b0f2e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.848785 895 tablet_service.cc:1468] Processing CreateTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca (DEFAULT_TABLE table=default.Table_963d2af04f874c58a54abb3ac76416a6 [id=c22f2e9e8d084f2089def95c26c2b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.849154 894 tablet_service.cc:1468] Processing CreateTablet for tablet b966a9c2749640808cf97059845d3b44 (DEFAULT_TABLE table=default.Table_ca6aba8125c9496d86aa4dff2eed2591 [id=d4e47fc75da249138358461212d80e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.849499 892 tablet_service.cc:1468] Processing CreateTablet for tablet 8453efdd649747a78c94eaa82fcde378 (DEFAULT_TABLE table=default.Table_bf8dc25dca7744e5b2ba3b14f1713b89 [id=00acceb7e9af4dc8b6fe843e436bf156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.849642 891 tablet_service.cc:1468] Processing CreateTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 (DEFAULT_TABLE table=default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.850378 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9af16f666fc6479ab67e449d1e962f8f (DEFAULT_TABLE table=default.Table_55740e7f990f470d86e7e7fc429b6d79 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.850697 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6690e00dfaa4d12be27e2fed9d835ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.856489 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8453efdd649747a78c94eaa82fcde378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.860345 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17dfc03d30ed40a498cae5efdc5445c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.864768 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b966a9c2749640808cf97059845d3b44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.865568 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b966a9c2749640808cf97059845d3b44
I20250629 01:51:38.867391 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:38.867798 890 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:38.868422 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9af16f666fc6479ab67e449d1e962f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.874275 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:38.883668 1155 tablet_bootstrap.cc:492] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:38.886780 1155 tablet_bootstrap.cc:654] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:38.887830 1155 log.cc:826] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:38.892160 1154 tablet_bootstrap.cc:492] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:38.892329 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b966a9c2749640808cf97059845d3b44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b966a9c2749640808cf97059845d3b44 already in progress: creating tablet
I20250629 01:51:38.892915 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c440da96423948c4bc188695d63152cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.893537 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c440da96423948c4bc188695d63152cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.901732 1155 tablet_bootstrap.cc:492] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:38.901993 1155 ts_tablet_manager.cc:1397] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.002s
I20250629 01:51:38.902444 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_20f0297464ef4a569b4432d1fc39b500"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.903013 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20f0297464ef4a569b4432d1fc39b500 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.904323 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_40480b1b197749ddbe37aac251167ba2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:38.904640 1159 tablet_bootstrap.cc:492] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:38.904801 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40480b1b197749ddbe37aac251167ba2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:38.906512 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b966a9c2749640808cf97059845d3b44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b966a9c2749640808cf97059845d3b44 already in progress: creating tablet
I20250629 01:51:38.907613 1159 tablet_bootstrap.cc:654] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:38.909768 1155 raft_consensus.cc:357] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:38.910085 1155 raft_consensus.cc:383] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:38.910192 1155 raft_consensus.cc:738] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:38.910799 1155 consensus_queue.cc:260] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:38.917058 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d17fb5b9d5da45f2a5d30831360a1cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.917693 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d17fb5b9d5da45f2a5d30831360a1cd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.921142 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2258f581be664efdbbc38536c0e60672"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.921674 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2258f581be664efdbbc38536c0e60672 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.921675 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 8453efdd649747a78c94eaa82fcde378 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:38.922876 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e3f08876e6774162a6aff102cc0e7bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.923383 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3f08876e6774162a6aff102cc0e7bc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.924052 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_34e0e15202b14e3bb0081120245e2033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.924582 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34e0e15202b14e3bb0081120245e2033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.924876 1155 ts_tablet_manager.cc:1428] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.023s user 0.008s sys 0.003s
I20250629 01:51:38.925457 1155 tablet_bootstrap.cc:492] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:38.927139 1155 tablet_bootstrap.cc:654] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:38.928468 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3d95402a158a4d9db8d3d262a4cdfd46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.928876 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d95402a158a4d9db8d3d262a4cdfd46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.931806 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4dccf19638764ac494fc06ce3a7ae4bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.932240 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dccf19638764ac494fc06ce3a7ae4bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.937430 1155 tablet_bootstrap.cc:492] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:38.937585 1155 ts_tablet_manager.cc:1397] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20250629 01:51:38.938479 1155 raft_consensus.cc:357] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:38.938663 1155 raft_consensus.cc:383] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:38.938741 1155 raft_consensus.cc:738] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:38.938989 1155 consensus_queue.cc:260] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:38.939656 1155 ts_tablet_manager.cc:1428] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:38.939920 1155 tablet_bootstrap.cc:492] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:38.941555 1155 tablet_bootstrap.cc:654] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:38.946898 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b966a9c2749640808cf97059845d3b44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b966a9c2749640808cf97059845d3b44 already in progress: creating tablet
I20250629 01:51:38.948063 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ddd0f8f961e244d49f7dd4136a7d2325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:38.948585 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd0f8f961e244d49f7dd4136a7d2325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:38.953341 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:38.953609 1156 tablet_replica.cc:331] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:38.953960 1156 raft_consensus.cc:2241] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:38.954296 1156 raft_consensus.cc:2270] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:38.955544 1156 ts_tablet_manager.cc:1905] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:38.959023 751 tablet_service.cc:1468] Processing CreateTablet for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (DEFAULT_TABLE table=default.Table_40480b1b197749ddbe37aac251167ba2 [id=ac80888442774af495a53bba44dfa763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.959578 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88031f8d8cbc44e4b8c53d9cd5360552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.960901 455 master_service.cc:948] ReplaceTablet: received request to replace tablet f6690e00dfaa4d12be27e2fed9d835ca from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:38.961385 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (DEFAULT_TABLE table=default.Table_40480b1b197749ddbe37aac251167ba2 [id=ac80888442774af495a53bba44dfa763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.961776 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88031f8d8cbc44e4b8c53d9cd5360552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.963383 1156 ts_tablet_manager.cc:1918] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:38.963515 1156 log.cc:1199] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b966a9c2749640808cf97059845d3b44
I20250629 01:51:38.963863 1156 ts_tablet_manager.cc:1939] T b966a9c2749640808cf97059845d3b44 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:38.964485 450 master_service.cc:948] ReplaceTablet: received request to replace tablet f6690e00dfaa4d12be27e2fed9d835ca from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:38.965164 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b966a9c2749640808cf97059845d3b44 (table default.Table_ca6aba8125c9496d86aa4dff2eed2591 [id=d4e47fc75da249138358461212d80e11]) successfully deleted
I20250629 01:51:38.971280 1159 log.cc:826] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:38.971885 1154 tablet_bootstrap.cc:654] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:38.974112 1154 log.cc:826] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:38.974341 894 tablet_service.cc:1468] Processing CreateTablet for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (DEFAULT_TABLE table=default.Table_40480b1b197749ddbe37aac251167ba2 [id=ac80888442774af495a53bba44dfa763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:38.974740 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88031f8d8cbc44e4b8c53d9cd5360552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:38.980759 1159 tablet_bootstrap.cc:492] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:38.981107 1159 ts_tablet_manager.cc:1397] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.005s
I20250629 01:51:38.996969 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_40480b1b197749ddbe37aac251167ba2" } new_table_name: "default.Table_66f7cc85f9864d11a6f9cb1d64bfa69d" modify_external_catalogs: true
I20250629 01:51:38.999644 1154 tablet_bootstrap.cc:492] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:38.999872 1154 ts_tablet_manager.cc:1397] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.002s
I20250629 01:51:39.000484 1155 tablet_bootstrap.cc:492] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.000648 1155 ts_tablet_manager.cc:1397] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.061s user 0.018s sys 0.028s
I20250629 01:51:39.001320 1155 raft_consensus.cc:357] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.001534 1155 raft_consensus.cc:383] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.001663 1155 raft_consensus.cc:738] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.001871 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 17dfc03d30ed40a498cae5efdc5445c7 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.001895 1155 consensus_queue.cc:260] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.002413 1155 ts_tablet_manager.cc:1428] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:39.003437 1159 raft_consensus.cc:357] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.003749 1159 raft_consensus.cc:383] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.003859 1159 raft_consensus.cc:738] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.006325 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55740e7f990f470d86e7e7fc429b6d79" } new_table_name: "default.Table_e73a4b091c874ab0b2f5d9423f300110" modify_external_catalogs: true
I20250629 01:51:39.006323 1159 consensus_queue.cc:260] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.007496 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7017fd5ca5754ea0b74a91ffa9017025" } modify_external_catalogs: true
I20250629 01:51:39.007658 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7017fd5ca5754ea0b74a91ffa9017025" } modify_external_catalogs: true
I20250629 01:51:39.009137 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9af16f666fc6479ab67e449d1e962f8f (table default.Table_55740e7f990f470d86e7e7fc429b6d79 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]). Expected version 1 got 0
I20250629 01:51:39.009730 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c440da96423948c4bc188695d63152cc" } modify_external_catalogs: true
I20250629 01:51:39.009917 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c440da96423948c4bc188695d63152cc" } modify_external_catalogs: true
I20250629 01:51:39.011451 1154 raft_consensus.cc:357] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.011741 1154 raft_consensus.cc:383] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.011842 1154 raft_consensus.cc:738] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.012511 1154 consensus_queue.cc:260] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.015451 1159 ts_tablet_manager.cc:1428] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.034s user 0.007s sys 0.005s
I20250629 01:51:39.017838 1159 tablet_bootstrap.cc:492] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.019273 1159 tablet_bootstrap.cc:654] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.022891 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8453efdd649747a78c94eaa82fcde378
I20250629 01:51:39.023377 453 catalog_manager.cc:6652] ReplaceTablet: tablet 8453efdd649747a78c94eaa82fcde378 deleted and replaced by tablet 1185e85345664dfd92c134c5552e7d48
I20250629 01:51:39.024354 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f6690e00dfaa4d12be27e2fed9d835ca
I20250629 01:51:39.024753 450 catalog_manager.cc:6652] ReplaceTablet: tablet f6690e00dfaa4d12be27e2fed9d835ca deleted and replaced by tablet d237b9f962024ecb967f86a5730a5039
I20250629 01:51:39.025064 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bf8dc25dca7744e5b2ba3b14f1713b89" } new_table_name: "default.Table_07d0ae14810e4c31b91252f2b672ef5d" modify_external_catalogs: true
I20250629 01:51:39.026942 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_963d2af04f874c58a54abb3ac76416a6" } new_table_name: "default.Table_ade22ca69a16455b9d8ded9bb0a060dc" modify_external_catalogs: true
W20250629 01:51:39.027580 825 tablet.cc:2378] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250629 01:51:39.029682 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.029991 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.030136 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca with delete_type TABLET_DATA_DELETED (replaced by tablet d237b9f962024ecb967f86a5730a5039) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.030290 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca with delete_type TABLET_DATA_DELETED (replaced by tablet d237b9f962024ecb967f86a5730a5039) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:39.031049 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9af16f666fc6479ab67e449d1e962f8f consensus state has no leader
I20250629 01:51:39.031385 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d17fb5b9d5da45f2a5d30831360a1cd0" } new_table_name: "default.Table_04985f57403a42598f1c7ef473177a7b" modify_external_catalogs: true
I20250629 01:51:39.032766 451 master_service.cc:948] ReplaceTablet: received request to replace tablet b966a9c2749640808cf97059845d3b44 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.033594 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet da27c51828684053b4f777739b0f2e51
W20250629 01:51:39.033879 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8453efdd649747a78c94eaa82fcde378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8453efdd649747a78c94eaa82fcde378 already in progress: creating tablet
W20250629 01:51:39.034098 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f6690e00dfaa4d12be27e2fed9d835ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6690e00dfaa4d12be27e2fed9d835ca already in progress: creating tablet
I20250629 01:51:39.034137 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_34e0e15202b14e3bb0081120245e2033" } modify_external_catalogs: true
I20250629 01:51:39.034255 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_34e0e15202b14e3bb0081120245e2033" } modify_external_catalogs: true
I20250629 01:51:39.034991 755 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.035326 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.035876 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet da27c51828684053b4f777739b0f2e51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da27c51828684053b4f777739b0f2e51 already in progress: creating tablet
I20250629 01:51:39.039008 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3d95402a158a4d9db8d3d262a4cdfd46" } modify_external_catalogs: true
I20250629 01:51:39.039203 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3d95402a158a4d9db8d3d262a4cdfd46" } modify_external_catalogs: true
I20250629 01:51:39.040516 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 17dfc03d30ed40a498cae5efdc5445c7
I20250629 01:51:39.040926 454 catalog_manager.cc:6652] ReplaceTablet: tablet 17dfc03d30ed40a498cae5efdc5445c7 deleted and replaced by tablet 20958c5e88454e129aa8efefb63b7f14
I20250629 01:51:39.041509 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.041899 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.042445 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 17dfc03d30ed40a498cae5efdc5445c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17dfc03d30ed40a498cae5efdc5445c7 already in progress: creating tablet
W20250629 01:51:39.042570 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88031f8d8cbc44e4b8c53d9cd5360552 consensus state has no leader
I20250629 01:51:39.044698 1164 raft_consensus.cc:491] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:39.044906 1164 raft_consensus.cc:513] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.046732 1164 leader_election.cc:290] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:39.049887 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 595ac9a136ad493a9a820755e1c938cd
W20250629 01:51:39.049969 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d237b9f962024ecb967f86a5730a5039 has no consensus state
W20250629 01:51:39.050267 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ab05d7564a8476c9f56c06687d11f36 has no consensus state
W20250629 01:51:39.051388 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1185e85345664dfd92c134c5552e7d48 has no consensus state
I20250629 01:51:39.057627 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.058081 894 tablet_service.cc:1515] Processing DeleteTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca with delete_type TABLET_DATA_DELETED (replaced by tablet d237b9f962024ecb967f86a5730a5039) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.058877 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8453efdd649747a78c94eaa82fcde378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8453efdd649747a78c94eaa82fcde378 already in progress: creating tablet
I20250629 01:51:39.059019 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet b171b12d103a480280896a3bb6da02dd
I20250629 01:51:39.059026 896 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.059106 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f6690e00dfaa4d12be27e2fed9d835ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6690e00dfaa4d12be27e2fed9d835ca already in progress: creating tablet
I20250629 01:51:39.058907 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.060153 1157 tablet_replica.cc:331] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.060484 1157 raft_consensus.cc:2241] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.060781 1157 raft_consensus.cc:2270] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:39.061017 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet da27c51828684053b4f777739b0f2e51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da27c51828684053b4f777739b0f2e51 already in progress: creating tablet
I20250629 01:51:39.061180 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.062144 1157 ts_tablet_manager.cc:1905] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.063602 1098 heartbeater.cc:499] Master 127.0.104.62:34747 was elected leader, sending a full tablet report...
I20250629 01:51:39.069000 1157 ts_tablet_manager.cc:1918] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.069193 1157 log.cc:1199] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b966a9c2749640808cf97059845d3b44
I20250629 01:51:39.069562 1157 ts_tablet_manager.cc:1939] T b966a9c2749640808cf97059845d3b44 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.070729 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20f0297464ef4a569b4432d1fc39b500" } modify_external_catalogs: true
I20250629 01:51:39.070878 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20f0297464ef4a569b4432d1fc39b500" } modify_external_catalogs: true
I20250629 01:51:39.071126 1154 ts_tablet_manager.cc:1428] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.071s user 0.018s sys 0.005s
I20250629 01:51:39.076918 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_66f7cc85f9864d11a6f9cb1d64bfa69d [id=ac80888442774af495a53bba44dfa763]). Expected version 1 got 0
I20250629 01:51:39.077654 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu pid 563
I20250629 01:51:39.077704 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9af16f666fc6479ab67e449d1e962f8f (table default.Table_e73a4b091c874ab0b2f5d9423f300110 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]). Expected version 2 got 0
I20250629 01:51:39.078200 1154 tablet_bootstrap.cc:492] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.078831 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250629 01:51:39.079763 1156 tablet_replica.cc:331] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.079939 1156 raft_consensus.cc:2241] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.080056 1156 raft_consensus.cc:2270] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.080236 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ab59bcc265cd41a7852b98dc84b34aff (DEFAULT_TABLE table=default.Table_2258f581be664efdbbc38536c0e60672 [id=affc7a0fc0c44e73bc261978f6782024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.080602 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59bcc265cd41a7852b98dc84b34aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.080628 1156 ts_tablet_manager.cc:1905] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.084844 755 tablet_service.cc:1468] Processing CreateTablet for tablet ab59bcc265cd41a7852b98dc84b34aff (DEFAULT_TABLE table=default.Table_2258f581be664efdbbc38536c0e60672 [id=affc7a0fc0c44e73bc261978f6782024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.085222 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59bcc265cd41a7852b98dc84b34aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.085573 1154 tablet_bootstrap.cc:654] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.086690 1156 ts_tablet_manager.cc:1918] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.086858 1156 log.cc:1199] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8453efdd649747a78c94eaa82fcde378
I20250629 01:51:39.086885 1159 tablet_bootstrap.cc:492] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.087035 1159 ts_tablet_manager.cc:1397] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.011s
I20250629 01:51:39.087318 1156 ts_tablet_manager.cc:1939] T 8453efdd649747a78c94eaa82fcde378 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.087630 1159 raft_consensus.cc:357] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.087805 1159 raft_consensus.cc:383] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.087882 1159 raft_consensus.cc:738] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.088145 1159 consensus_queue.cc:260] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.088567 1159 ts_tablet_manager.cc:1428] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:39.088614 1155 tablet_bootstrap.cc:492] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:39.092278 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9af16f666fc6479ab67e449d1e962f8f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:39.092753 1052 raft_consensus.cc:2466] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:39.092676 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9af16f666fc6479ab67e449d1e962f8f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:39.093012 1159 tablet_bootstrap.cc:492] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.093331 710 leader_election.cc:304] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:39.092761 1154 tablet_bootstrap.cc:492] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.093639 1164 raft_consensus.cc:2802] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:39.093758 1164 raft_consensus.cc:491] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:39.093816 1154 ts_tablet_manager.cc:1397] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250629 01:51:39.093851 1164 raft_consensus.cc:3058] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:39.094713 1154 raft_consensus.cc:357] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.094843 1159 tablet_bootstrap.cc:654] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.094913 1154 raft_consensus.cc:383] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.094987 1154 raft_consensus.cc:738] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.095268 1154 consensus_queue.cc:260] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.095747 1154 ts_tablet_manager.cc:1428] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:39.095891 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b966a9c2749640808cf97059845d3b44 (table default.Table_ca6aba8125c9496d86aa4dff2eed2591 [id=d4e47fc75da249138358461212d80e11]) successfully deleted
W20250629 01:51:39.096474 712 leader_election.cc:343] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:39.096504 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 17dfc03d30ed40a498cae5efdc5445c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17dfc03d30ed40a498cae5efdc5445c7 already in progress: creating tablet
I20250629 01:51:39.098105 894 tablet_service.cc:1468] Processing CreateTablet for tablet ab59bcc265cd41a7852b98dc84b34aff (DEFAULT_TABLE table=default.Table_2258f581be664efdbbc38536c0e60672 [id=affc7a0fc0c44e73bc261978f6782024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.098489 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59bcc265cd41a7852b98dc84b34aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.098975 1154 tablet_bootstrap.cc:492] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.100775 1154 tablet_bootstrap.cc:654] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:39.100915 1102 tablet.cc:2378] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250629 01:51:39.104660 1155 tablet_bootstrap.cc:654] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.108227 1164 raft_consensus.cc:513] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.109341 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9af16f666fc6479ab67e449d1e962f8f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:39.109508 1052 raft_consensus.cc:3058] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:39.112164 1164 leader_election.cc:290] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:39.112721 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9af16f666fc6479ab67e449d1e962f8f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:39.113214 712 leader_election.cc:343] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:39.113468 1052 raft_consensus.cc:2466] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:39.113827 710 leader_election.cc:304] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:39.114377 1164 raft_consensus.cc:2802] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:39.120374 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d6fd2102b5b84ae4ae6d58f2a577ee69
I20250629 01:51:39.120885 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3963492a9da142409d4958813415dee6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.121218 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.122640 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3963492a9da142409d4958813415dee6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.124536 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8453efdd649747a78c94eaa82fcde378 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]) successfully deleted
I20250629 01:51:39.124979 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.125576 442 catalog_manager.cc:4895] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 because the tablet was not found. No further retry: Not found: Tablet not found: d6fd2102b5b84ae4ae6d58f2a577ee69
I20250629 01:51:39.125910 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.126261 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab05d7564a8476c9f56c06687d11f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.126689 896 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.126982 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab05d7564a8476c9f56c06687d11f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.128372 895 tablet_service.cc:1468] Processing CreateTablet for tablet ee70cbe4d80e41b99234f1e1b001434b (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.128661 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee70cbe4d80e41b99234f1e1b001434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.132968 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0b710052005c4a2aaade1dc60409aef3 (DEFAULT_TABLE table=default.Table_c440da96423948c4bc188695d63152cc [id=64a5b540bb01488c92bb18807a03e2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.133262 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b710052005c4a2aaade1dc60409aef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.135561 890 tablet_service.cc:1468] Processing CreateTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 (DEFAULT_TABLE table=default.Table_20f0297464ef4a569b4432d1fc39b500 [id=dc5b44952f4a43c88249de26b060f934]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.135974 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd2102b5b84ae4ae6d58f2a577ee69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.136327 893 tablet_service.cc:1468] Processing CreateTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf (DEFAULT_TABLE table=default.Table_4dccf19638764ac494fc06ce3a7ae4bf [id=2a49e00329f14706ae427516ea2072c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.136636 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88fdf1f8bb96449ba472990092cfe8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.140625 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9af16f666fc6479ab67e449d1e962f8f (table default.Table_e73a4b091c874ab0b2f5d9423f300110 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]). Expected version 2 got 0
I20250629 01:51:39.142053 751 tablet_service.cc:1468] Processing CreateTablet for tablet 0b710052005c4a2aaade1dc60409aef3 (DEFAULT_TABLE table=default.Table_c440da96423948c4bc188695d63152cc [id=64a5b540bb01488c92bb18807a03e2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.142374 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b710052005c4a2aaade1dc60409aef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.144908 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_adb2b0b91c94485fa3ede26987189123"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.145349 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adb2b0b91c94485fa3ede26987189123 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.145951 752 tablet_service.cc:1468] Processing CreateTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.146286 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab05d7564a8476c9f56c06687d11f36. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.152588 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 because the tablet was not found. No further retry: Not found: Tablet not found: d6fd2102b5b84ae4ae6d58f2a577ee69
I20250629 01:51:39.152583 1032 tablet_service.cc:1468] Processing CreateTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 (DEFAULT_TABLE table=default.Table_20f0297464ef4a569b4432d1fc39b500 [id=dc5b44952f4a43c88249de26b060f934]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:39.152824 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f6690e00dfaa4d12be27e2fed9d835ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6690e00dfaa4d12be27e2fed9d835ca already in progress: creating tablet
I20250629 01:51:39.152922 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd2102b5b84ae4ae6d58f2a577ee69. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.152971 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet da27c51828684053b4f777739b0f2e51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da27c51828684053b4f777739b0f2e51 already in progress: creating tablet
W20250629 01:51:39.153124 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 17dfc03d30ed40a498cae5efdc5445c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17dfc03d30ed40a498cae5efdc5445c7 already in progress: creating tablet
I20250629 01:51:39.154382 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.154770 757 tablet_service.cc:1468] Processing CreateTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 (DEFAULT_TABLE table=default.Table_20f0297464ef4a569b4432d1fc39b500 [id=dc5b44952f4a43c88249de26b060f934]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.155041 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6fd2102b5b84ae4ae6d58f2a577ee69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.159178 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0803b366c30642e681bff17044760ace"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.159576 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0803b366c30642e681bff17044760ace in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.159682 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6385bcc046a24e29b29108aa7a0e14d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.160085 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6385bcc046a24e29b29108aa7a0e14d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.160514 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.160718 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca with delete_type TABLET_DATA_DELETED (replaced by tablet d237b9f962024ecb967f86a5730a5039) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.161051 756 tablet_service.cc:1468] Processing CreateTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf (DEFAULT_TABLE table=default.Table_4dccf19638764ac494fc06ce3a7ae4bf [id=2a49e00329f14706ae427516ea2072c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.161216 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ee70cbe4d80e41b99234f1e1b001434b (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.161352 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88fdf1f8bb96449ba472990092cfe8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.161496 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee70cbe4d80e41b99234f1e1b001434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.164240 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.164511 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.166693 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b1e97058cbf34ff4b34ec7ef01e2b47b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.167197 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1e97058cbf34ff4b34ec7ef01e2b47b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.170804 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8453efdd649747a78c94eaa82fcde378 because the tablet was not found. No further retry: Not found: Tablet not found: 8453efdd649747a78c94eaa82fcde378
I20250629 01:51:39.170980 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1e0ef1bcacd9498898d0c997cfaf05d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.171481 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e0ef1bcacd9498898d0c997cfaf05d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.177959 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.178169 1157 tablet_replica.cc:331] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.178124 752 tablet_service.cc:1468] Processing CreateTablet for tablet ee70cbe4d80e41b99234f1e1b001434b (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.178318 1157 raft_consensus.cc:2241] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.178406 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee70cbe4d80e41b99234f1e1b001434b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.178489 1157 raft_consensus.cc:2270] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.183686 755 tablet_service.cc:1515] Processing DeleteTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.184244 1157 ts_tablet_manager.cc:1905] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.184629 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.185209 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf (DEFAULT_TABLE table=default.Table_4dccf19638764ac494fc06ce3a7ae4bf [id=2a49e00329f14706ae427516ea2072c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.185771 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88fdf1f8bb96449ba472990092cfe8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.188241 1158 tablet_replica.cc:331] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:39.188226 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b966a9c2749640808cf97059845d3b44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b966a9c2749640808cf97059845d3b44 already in progress: creating tablet
W20250629 01:51:39.188467 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet da27c51828684053b4f777739b0f2e51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da27c51828684053b4f777739b0f2e51 already in progress: creating tablet
I20250629 01:51:39.188606 1158 raft_consensus.cc:2241] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:39.188664 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6fd2102b5b84ae4ae6d58f2a577ee69 already in progress: creating tablet
I20250629 01:51:39.188946 1158 raft_consensus.cc:2270] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.190373 1158 ts_tablet_manager.cc:1905] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.191327 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0b710052005c4a2aaade1dc60409aef3
I20250629 01:51:39.193529 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b710052005c4a2aaade1dc60409aef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.193712 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b710052005c4a2aaade1dc60409aef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.196370 894 tablet_service.cc:1468] Processing CreateTablet for tablet 0b710052005c4a2aaade1dc60409aef3 (DEFAULT_TABLE table=default.Table_c440da96423948c4bc188695d63152cc [id=64a5b540bb01488c92bb18807a03e2b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.196699 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b710052005c4a2aaade1dc60409aef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.201313 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_abbad35fa11f4103ad64ef1ea77dbd65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.202008 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abbad35fa11f4103ad64ef1ea77dbd65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.203143 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_df03454cb2674352937e6fa8a63f5141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.203547 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df03454cb2674352937e6fa8a63f5141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.203956 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d237b9f962024ecb967f86a5730a5039 (DEFAULT_TABLE table=default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.204285 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d237b9f962024ecb967f86a5730a5039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.204607 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 20958c5e88454e129aa8efefb63b7f14 (DEFAULT_TABLE table=default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.205039 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20958c5e88454e129aa8efefb63b7f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.209400 751 tablet_service.cc:1468] Processing CreateTablet for tablet d237b9f962024ecb967f86a5730a5039 (DEFAULT_TABLE table=default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.209717 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d237b9f962024ecb967f86a5730a5039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.212357 753 tablet_service.cc:1468] Processing CreateTablet for tablet 1185e85345664dfd92c134c5552e7d48 (DEFAULT_TABLE table=default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.212483 896 tablet_service.cc:1468] Processing CreateTablet for tablet 1185e85345664dfd92c134c5552e7d48 (DEFAULT_TABLE table=default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.212658 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1185e85345664dfd92c134c5552e7d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.212754 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1185e85345664dfd92c134c5552e7d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.216328 893 tablet_service.cc:1468] Processing CreateTablet for tablet d237b9f962024ecb967f86a5730a5039 (DEFAULT_TABLE table=default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.216615 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d237b9f962024ecb967f86a5730a5039. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.220355 890 tablet_service.cc:1468] Processing CreateTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (DEFAULT_TABLE table=default.Table_ddd0f8f961e244d49f7dd4136a7d2325 [id=9ff841a004c344c688f5abf6b59b5dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.220647 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.222604 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a3096bcff0004df0a66caa7d3fbc796f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.222961 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3096bcff0004df0a66caa7d3fbc796f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.225310 757 tablet_service.cc:1468] Processing CreateTablet for tablet 20958c5e88454e129aa8efefb63b7f14 (DEFAULT_TABLE table=default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.225706 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20958c5e88454e129aa8efefb63b7f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.234805 896 tablet_service.cc:1468] Processing CreateTablet for tablet 0d46683318dc4c04a617a99f3e620734 (DEFAULT_TABLE table=default.Table_1e0ef1bcacd9498898d0c997cfaf05d4 [id=f7df5c11e9a84f61b749ebf60a50e6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.235330 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d46683318dc4c04a617a99f3e620734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.236335 1157 ts_tablet_manager.cc:1918] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.236389 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ddd0f8f961e244d49f7dd4136a7d2325" } new_table_name: "default.Table_fef90accef4548ee9603f9b54d9a8a84" modify_external_catalogs: true
I20250629 01:51:39.236588 1157 log.cc:1199] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/17dfc03d30ed40a498cae5efdc5445c7
I20250629 01:51:39.236981 1157 ts_tablet_manager.cc:1939] T 17dfc03d30ed40a498cae5efdc5445c7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.237749 891 tablet_service.cc:1468] Processing CreateTablet for tablet 57f7f71ab5e54d249141c89d9e806111 (DEFAULT_TABLE table=default.Table_6385bcc046a24e29b29108aa7a0e14d3 [id=60094a0a7c134476a2a4e44d8555c5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.237871 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 17dfc03d30ed40a498cae5efdc5445c7 (table default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]) successfully deleted
I20250629 01:51:39.238050 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f7f71ab5e54d249141c89d9e806111. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.238355 442 catalog_manager.cc:4895] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0b710052005c4a2aaade1dc60409aef3 because the tablet was not found. No further retry: Not found: Tablet not found: 0b710052005c4a2aaade1dc60409aef3
I20250629 01:51:39.238600 756 tablet_service.cc:1468] Processing CreateTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (DEFAULT_TABLE table=default.Table_ddd0f8f961e244d49f7dd4136a7d2325 [id=9ff841a004c344c688f5abf6b59b5dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.238984 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.240775 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e73a4b091c874ab0b2f5d9423f300110" } modify_external_catalogs: true
I20250629 01:51:39.240924 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e73a4b091c874ab0b2f5d9423f300110" } modify_external_catalogs: true
I20250629 01:51:39.241405 892 tablet_service.cc:1468] Processing CreateTablet for tablet dc6f34d6b31849e190ebfaf1a273d29c (DEFAULT_TABLE table=default.Table_3963492a9da142409d4958813415dee6 [id=c8577f19a8e04dc090d81e821bf6d36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.242190 453 master_service.cc:948] ReplaceTablet: received request to replace tablet ab59bcc265cd41a7852b98dc84b34aff from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.242221 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6f34d6b31849e190ebfaf1a273d29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.242455 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b710052005c4a2aaade1dc60409aef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.243978 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1e0ef1bcacd9498898d0c997cfaf05d4" } new_table_name: "default.Table_45d1774983d248aeab1a0353a2d1ece2" modify_external_catalogs: true
I20250629 01:51:39.244544 1158 ts_tablet_manager.cc:1918] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.244707 1158 log.cc:1199] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f6690e00dfaa4d12be27e2fed9d835ca
I20250629 01:51:39.245576 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3f1372c29f2b4177af51a749a873b87c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.245867 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0b710052005c4a2aaade1dc60409aef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b710052005c4a2aaade1dc60409aef3 already in progress: creating tablet
I20250629 01:51:39.245898 1158 ts_tablet_manager.cc:1939] T f6690e00dfaa4d12be27e2fed9d835ca P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:39.245977 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f1372c29f2b4177af51a749a873b87c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.246730 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 (DEFAULT_TABLE table=default.Table_b1e97058cbf34ff4b34ec7ef01e2b47b [id=2819c6ba47aa436a9ddd9a72f956c96e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.247064 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267d9a050fa2414d87a04ba3c91dea03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.247537 454 master_service.cc:948] ReplaceTablet: received request to replace tablet f6690e00dfaa4d12be27e2fed9d835ca from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.249606 896 tablet_service.cc:1468] Processing CreateTablet for tablet b3c896a37d9749c5a46bfc7cf7c87911 (DEFAULT_TABLE table=default.Table_adb2b0b91c94485fa3ede26987189123 [id=50b11e4f7e8c49de8a781b273a593358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.249912 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c896a37d9749c5a46bfc7cf7c87911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.249892 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 57f7f71ab5e54d249141c89d9e806111 (DEFAULT_TABLE table=default.Table_6385bcc046a24e29b29108aa7a0e14d3 [id=60094a0a7c134476a2a4e44d8555c5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.250196 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f7f71ab5e54d249141c89d9e806111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.253641 1032 tablet_service.cc:1468] Processing CreateTablet for tablet dc6f34d6b31849e190ebfaf1a273d29c (DEFAULT_TABLE table=default.Table_3963492a9da142409d4958813415dee6 [id=c8577f19a8e04dc090d81e821bf6d36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.253930 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6f34d6b31849e190ebfaf1a273d29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.255960 1024 tablet_service.cc:1468] Processing CreateTablet for tablet b3c896a37d9749c5a46bfc7cf7c87911 (DEFAULT_TABLE table=default.Table_adb2b0b91c94485fa3ede26987189123 [id=50b11e4f7e8c49de8a781b273a593358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.256433 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c896a37d9749c5a46bfc7cf7c87911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.256981 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2258f581be664efdbbc38536c0e60672" } new_table_name: "default.Table_7724653350cb485cb25696194e364d3f" modify_external_catalogs: true
I20250629 01:51:39.257326 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_df03454cb2674352937e6fa8a63f5141" } new_table_name: "default.Table_8603f3907bdd482ea98c45f0c5cd7903" modify_external_catalogs: true
I20250629 01:51:39.257530 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f6690e00dfaa4d12be27e2fed9d835ca (table default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]) successfully deleted
W20250629 01:51:39.257730 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8453efdd649747a78c94eaa82fcde378 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8453efdd649747a78c94eaa82fcde378 already in progress: creating tablet
W20250629 01:51:39.257903 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 17dfc03d30ed40a498cae5efdc5445c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17dfc03d30ed40a498cae5efdc5445c7 already in progress: creating tablet
W20250629 01:51:39.258122 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0b710052005c4a2aaade1dc60409aef3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b710052005c4a2aaade1dc60409aef3 already in progress: creating tablet
I20250629 01:51:39.259250 452 master_service.cc:948] ReplaceTablet: received request to replace tablet f6690e00dfaa4d12be27e2fed9d835ca from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.259760 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b1e97058cbf34ff4b34ec7ef01e2b47b" } new_table_name: "default.Table_8c79e92b6cfe429088a6b6662f46412c" modify_external_catalogs: true
I20250629 01:51:39.263772 895 tablet_service.cc:1468] Processing CreateTablet for tablet 20958c5e88454e129aa8efefb63b7f14 (DEFAULT_TABLE table=default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.264142 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20958c5e88454e129aa8efefb63b7f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.269119 754 tablet_service.cc:1468] Processing CreateTablet for tablet b3c896a37d9749c5a46bfc7cf7c87911 (DEFAULT_TABLE table=default.Table_adb2b0b91c94485fa3ede26987189123 [id=50b11e4f7e8c49de8a781b273a593358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.269421 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c896a37d9749c5a46bfc7cf7c87911. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.271649 755 tablet_service.cc:1468] Processing CreateTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 (DEFAULT_TABLE table=default.Table_b1e97058cbf34ff4b34ec7ef01e2b47b [id=2819c6ba47aa436a9ddd9a72f956c96e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.271994 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267d9a050fa2414d87a04ba3c91dea03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.276177 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4dccf19638764ac494fc06ce3a7ae4bf" } modify_external_catalogs: true
I20250629 01:51:39.276327 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4dccf19638764ac494fc06ce3a7ae4bf" } modify_external_catalogs: true
I20250629 01:51:39.280189 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 7ab05d7564a8476c9f56c06687d11f36 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.280679 751 tablet_service.cc:1468] Processing CreateTablet for tablet 57f7f71ab5e54d249141c89d9e806111 (DEFAULT_TABLE table=default.Table_6385bcc046a24e29b29108aa7a0e14d3 [id=60094a0a7c134476a2a4e44d8555c5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.280994 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f7f71ab5e54d249141c89d9e806111. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.282765 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf consensus state has no leader
I20250629 01:51:39.282986 450 master_service.cc:948] ReplaceTablet: received request to replace tablet da27c51828684053b4f777739b0f2e51 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.283461 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ade22ca69a16455b9d8ded9bb0a060dc" } modify_external_catalogs: true
I20250629 01:51:39.283587 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ade22ca69a16455b9d8ded9bb0a060dc" } modify_external_catalogs: true
I20250629 01:51:39.284389 893 tablet_service.cc:1468] Processing CreateTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 (DEFAULT_TABLE table=default.Table_0803b366c30642e681bff17044760ace [id=0aeee3c29f1241c4b911b1a390686afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.284730 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3395b9c2d04a6f8a227630087d85d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.289404 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9af16f666fc6479ab67e449d1e962f8f
I20250629 01:51:39.290683 753 tablet_service.cc:1468] Processing CreateTablet for tablet dc6f34d6b31849e190ebfaf1a273d29c (DEFAULT_TABLE table=default.Table_3963492a9da142409d4958813415dee6 [id=c8577f19a8e04dc090d81e821bf6d36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.291041 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc6f34d6b31849e190ebfaf1a273d29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.292284 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ab59bcc265cd41a7852b98dc84b34aff
I20250629 01:51:39.292668 453 catalog_manager.cc:6652] ReplaceTablet: tablet ab59bcc265cd41a7852b98dc84b34aff deleted and replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3
I20250629 01:51:39.293969 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9af16f666fc6479ab67e449d1e962f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.294077 891 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59bcc265cd41a7852b98dc84b34aff with delete_type TABLET_DATA_DELETED (replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.294209 1156 tablet_replica.cc:331] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.294524 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9af16f666fc6479ab67e449d1e962f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.294163 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 1185e85345664dfd92c134c5552e7d48 (DEFAULT_TABLE table=default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.294263 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59bcc265cd41a7852b98dc84b34aff with delete_type TABLET_DATA_DELETED (replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.295334 890 tablet_service.cc:1468] Processing CreateTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 (DEFAULT_TABLE table=default.Table_b1e97058cbf34ff4b34ec7ef01e2b47b [id=2819c6ba47aa436a9ddd9a72f956c96e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.295643 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267d9a050fa2414d87a04ba3c91dea03. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.295905 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ab59bcc265cd41a7852b98dc84b34aff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab59bcc265cd41a7852b98dc84b34aff already in progress: creating tablet
I20250629 01:51:39.294565 754 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59bcc265cd41a7852b98dc84b34aff with delete_type TABLET_DATA_DELETED (replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.296547 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 9af16f666fc6479ab67e449d1e962f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.296932 1158 tablet_replica.cc:331] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:39.297320 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ab59bcc265cd41a7852b98dc84b34aff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab59bcc265cd41a7852b98dc84b34aff already in progress: creating tablet
I20250629 01:51:39.297564 1158 raft_consensus.cc:2241] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.297720 1158 raft_consensus.cc:2270] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.298408 1158 ts_tablet_manager.cc:1905] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:39.303624 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d46683318dc4c04a617a99f3e620734 consensus state has no leader
I20250629 01:51:39.305094 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 0d46683318dc4c04a617a99f3e620734 (DEFAULT_TABLE table=default.Table_1e0ef1bcacd9498898d0c997cfaf05d4 [id=f7df5c11e9a84f61b749ebf60a50e6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.305424 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d46683318dc4c04a617a99f3e620734. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.310736 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1185e85345664dfd92c134c5552e7d48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.310848 1158 ts_tablet_manager.cc:1918] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.312249 1158 log.cc:1199] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9af16f666fc6479ab67e449d1e962f8f
I20250629 01:51:39.312623 1158 ts_tablet_manager.cc:1939] T 9af16f666fc6479ab67e449d1e962f8f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:39.313598 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9af16f666fc6479ab67e449d1e962f8f (table default.Table_e73a4b091c874ab0b2f5d9423f300110 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]) successfully deleted
I20250629 01:51:39.307708 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (DEFAULT_TABLE table=default.Table_ddd0f8f961e244d49f7dd4136a7d2325 [id=9ff841a004c344c688f5abf6b59b5dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.314036 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.314442 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 (DEFAULT_TABLE table=default.Table_0803b366c30642e681bff17044760ace [id=0aeee3c29f1241c4b911b1a390686afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.314738 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3395b9c2d04a6f8a227630087d85d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.316421 757 tablet_service.cc:1468] Processing CreateTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 (DEFAULT_TABLE table=default.Table_0803b366c30642e681bff17044760ace [id=0aeee3c29f1241c4b911b1a390686afb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.316723 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b3395b9c2d04a6f8a227630087d85d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.322911 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0d46683318dc4c04a617a99f3e620734 (DEFAULT_TABLE table=default.Table_1e0ef1bcacd9498898d0c997cfaf05d4 [id=f7df5c11e9a84f61b749ebf60a50e6b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.323239 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d46683318dc4c04a617a99f3e620734. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.323781 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9af16f666fc6479ab67e449d1e962f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9af16f666fc6479ab67e449d1e962f8f already in progress: creating tablet
I20250629 01:51:39.326256 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_45d1774983d248aeab1a0353a2d1ece2" } modify_external_catalogs: true
I20250629 01:51:39.326411 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_45d1774983d248aeab1a0353a2d1ece2" } modify_external_catalogs: true
W20250629 01:51:39.335510 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95c14dacdd0d4156ba7b172edf255ae3 has no consensus state
W20250629 01:51:39.335752 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3de9ab31fc34ba88ac7616dff6e7d5e has no consensus state
W20250629 01:51:39.336316 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 267d9a050fa2414d87a04ba3c91dea03 consensus state has no leader
I20250629 01:51:39.341387 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d237b9f962024ecb967f86a5730a5039
I20250629 01:51:39.344991 895 tablet_service.cc:1515] Processing DeleteTablet for tablet d237b9f962024ecb967f86a5730a5039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.345515 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d237b9f962024ecb967f86a5730a5039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d237b9f962024ecb967f86a5730a5039 already in progress: creating tablet
I20250629 01:51:39.347630 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d237b9f962024ecb967f86a5730a5039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.348109 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d237b9f962024ecb967f86a5730a5039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d237b9f962024ecb967f86a5730a5039 already in progress: creating tablet
I20250629 01:51:39.351848 755 tablet_service.cc:1468] Processing CreateTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e (DEFAULT_TABLE table=default.Table_8603f3907bdd482ea98c45f0c5cd7903 [id=c95a278758c9465da9582afb968ec8f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.352175 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de9ab31fc34ba88ac7616dff6e7d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.353204 753 tablet_service.cc:1515] Processing DeleteTablet for tablet d237b9f962024ecb967f86a5730a5039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.356376 891 tablet_service.cc:1468] Processing CreateTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e (DEFAULT_TABLE table=default.Table_8603f3907bdd482ea98c45f0c5cd7903 [id=c95a278758c9465da9582afb968ec8f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.356750 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de9ab31fc34ba88ac7616dff6e7d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.356801 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (DEFAULT_TABLE table=default.Table_a3096bcff0004df0a66caa7d3fbc796f [id=006d3a71c3bc4d1190910bec2c905428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.357088 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d44151acb0d4bf4a630c086b1ac8ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.357626 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0d46683318dc4c04a617a99f3e620734
I20250629 01:51:39.357918 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7ab05d7564a8476c9f56c06687d11f36
I20250629 01:51:39.358296 455 catalog_manager.cc:6652] ReplaceTablet: tablet 7ab05d7564a8476c9f56c06687d11f36 deleted and replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4
I20250629 01:51:39.359791 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.359856 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.360052 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d46683318dc4c04a617a99f3e620734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.360989 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7ab05d7564a8476c9f56c06687d11f36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ab05d7564a8476c9f56c06687d11f36 already in progress: creating tablet
I20250629 01:51:39.361133 890 tablet_service.cc:1468] Processing CreateTablet for tablet ac26cba8a92945dd9acd10e8a76172ca (DEFAULT_TABLE table=default.Table_abbad35fa11f4103ad64ef1ea77dbd65 [id=29e100c61aaf4178b931f15a4467fc63]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:39.361246 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0d46683318dc4c04a617a99f3e620734 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d46683318dc4c04a617a99f3e620734 already in progress: creating tablet
I20250629 01:51:39.361481 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac26cba8a92945dd9acd10e8a76172ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.364665 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 88fdf1f8bb96449ba472990092cfe8cf
I20250629 01:51:39.364645 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ac26cba8a92945dd9acd10e8a76172ca (DEFAULT_TABLE table=default.Table_abbad35fa11f4103ad64ef1ea77dbd65 [id=29e100c61aaf4178b931f15a4467fc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.364981 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac26cba8a92945dd9acd10e8a76172ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.365571 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d46683318dc4c04a617a99f3e620734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.365805 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.366317 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7ab05d7564a8476c9f56c06687d11f36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ab05d7564a8476c9f56c06687d11f36 already in progress: creating tablet
W20250629 01:51:39.366847 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0d46683318dc4c04a617a99f3e620734 because the tablet was not found. No further retry: Not found: Tablet not found: 0d46683318dc4c04a617a99f3e620734
I20250629 01:51:39.367210 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.367730 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 88fdf1f8bb96449ba472990092cfe8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88fdf1f8bb96449ba472990092cfe8cf already in progress: creating tablet
W20250629 01:51:39.368254 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 88fdf1f8bb96449ba472990092cfe8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88fdf1f8bb96449ba472990092cfe8cf already in progress: creating tablet
I20250629 01:51:39.369010 895 tablet_service.cc:1468] Processing CreateTablet for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (DEFAULT_TABLE table=default.Table_a3096bcff0004df0a66caa7d3fbc796f [id=006d3a71c3bc4d1190910bec2c905428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.369570 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d44151acb0d4bf4a630c086b1ac8ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.370847 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.374296 756 tablet_service.cc:1468] Processing CreateTablet for tablet ac26cba8a92945dd9acd10e8a76172ca (DEFAULT_TABLE table=default.Table_abbad35fa11f4103ad64ef1ea77dbd65 [id=29e100c61aaf4178b931f15a4467fc63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.374840 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.375021 755 tablet_service.cc:1468] Processing CreateTablet for tablet b6688e9665f24e73b80802d47a80a2cd (DEFAULT_TABLE table=default.Table_3f1372c29f2b4177af51a749a873b87c [id=fadb6eabbead48b1a11795ffed0c4d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.375653 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 (DEFAULT_TABLE table=default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.375960 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c14dacdd0d4156ba7b172edf255ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.379173 1027 tablet_service.cc:1468] Processing CreateTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e (DEFAULT_TABLE table=default.Table_8603f3907bdd482ea98c45f0c5cd7903 [id=c95a278758c9465da9582afb968ec8f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.375296 753 tablet_service.cc:1468] Processing CreateTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 (DEFAULT_TABLE table=default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.375322 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac26cba8a92945dd9acd10e8a76172ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.380115 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6688e9665f24e73b80802d47a80a2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.380941 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de9ab31fc34ba88ac7616dff6e7d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.381095 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (DEFAULT_TABLE table=default.Table_a3096bcff0004df0a66caa7d3fbc796f [id=006d3a71c3bc4d1190910bec2c905428]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.381726 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d44151acb0d4bf4a630c086b1ac8ff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.383616 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.383782 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.383945 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b6688e9665f24e73b80802d47a80a2cd (DEFAULT_TABLE table=default.Table_3f1372c29f2b4177af51a749a873b87c [id=fadb6eabbead48b1a11795ffed0c4d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.384055 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a22af67c80541b38f2d4c121fdeb6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.384029 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a22af67c80541b38f2d4c121fdeb6e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.384361 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6688e9665f24e73b80802d47a80a2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.390115 1155 tablet_bootstrap.cc:492] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.390300 1155 ts_tablet_manager.cc:1397] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.302s user 0.019s sys 0.047s
I20250629 01:51:39.391053 1155 raft_consensus.cc:357] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.391412 1155 raft_consensus.cc:383] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.391515 1155 raft_consensus.cc:738] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.391769 1155 consensus_queue.cc:260] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.392427 1155 ts_tablet_manager.cc:1428] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:39.393020 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_036318a528c94f2fab8d77b241776f70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.393479 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_036318a528c94f2fab8d77b241776f70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.377360 1164 raft_consensus.cc:695] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:39.395433 1164 consensus_queue.cc:237] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.397557 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c14dacdd0d4156ba7b172edf255ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.398487 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_daa3abe221c44551848996265764f670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.398798 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4b11b5c3ee064217b3fcbeb9ad01293c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.398913 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_daa3abe221c44551848996265764f670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.399191 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.399875 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b8bdb1307ac249ac89d252814049090f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.400082 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c586a09ba29642f394d315d9b8ae758c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.400300 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8bdb1307ac249ac89d252814049090f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.400453 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c586a09ba29642f394d315d9b8ae758c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.401281 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a37f4562238b452abf51e6e3d547ca36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.401674 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a37f4562238b452abf51e6e3d547ca36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.379577 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d46683318dc4c04a617a99f3e620734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.411792 1154 tablet_bootstrap.cc:492] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.412521 1154 ts_tablet_manager.cc:1397] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.314s user 0.021s sys 0.046s
I20250629 01:51:39.413216 1154 raft_consensus.cc:357] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.413360 1154 raft_consensus.cc:383] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.413556 1154 raft_consensus.cc:738] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.413763 1154 consensus_queue.cc:260] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.416262 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_abbad35fa11f4103ad64ef1ea77dbd65" } modify_external_catalogs: true
I20250629 01:51:39.416414 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_abbad35fa11f4103ad64ef1ea77dbd65" } modify_external_catalogs: true
I20250629 01:51:39.416697 1154 ts_tablet_manager.cc:1428] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:39.421038 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 (DEFAULT_TABLE table=default.Table_036318a528c94f2fab8d77b241776f70 [id=d2da066ea4344ed4b3847246f6716583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.421300 1154 tablet_bootstrap.cc:492] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.421355 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc260579c0da4ed38d9b0168e469c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.421653 757 tablet_service.cc:1468] Processing CreateTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 (DEFAULT_TABLE table=default.Table_036318a528c94f2fab8d77b241776f70 [id=d2da066ea4344ed4b3847246f6716583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.421947 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc260579c0da4ed38d9b0168e469c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.424800 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.425012 893 tablet_service.cc:1468] Processing CreateTablet for tablet b6688e9665f24e73b80802d47a80a2cd (DEFAULT_TABLE table=default.Table_3f1372c29f2b4177af51a749a873b87c [id=fadb6eabbead48b1a11795ffed0c4d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.425365 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6688e9665f24e73b80802d47a80a2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.430094 894 tablet_service.cc:1468] Processing CreateTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 (DEFAULT_TABLE table=default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.430359 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95c14dacdd0d4156ba7b172edf255ae3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.431049 1158 tablet_replica.cc:331] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:39.431265 1158 raft_consensus.cc:2241] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.431468 1158 raft_consensus.cc:2270] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.432165 1158 ts_tablet_manager.cc:1905] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.432754 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7a50e6b289ed401092480b4d315d2840"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.433163 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a50e6b289ed401092480b4d315d2840 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.433665 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.436084 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e201257c73664abc93462558cfcdc261"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.436419 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a22af67c80541b38f2d4c121fdeb6e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.436518 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e201257c73664abc93462558cfcdc261 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.437810 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5a356c61f0ef4ddfba46aac25d270004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.438211 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a356c61f0ef4ddfba46aac25d270004 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.443089 1154 tablet_bootstrap.cc:654] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.445996 891 tablet_service.cc:1468] Processing CreateTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 (DEFAULT_TABLE table=default.Table_036318a528c94f2fab8d77b241776f70 [id=d2da066ea4344ed4b3847246f6716583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.446331 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc260579c0da4ed38d9b0168e469c1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.450698 890 tablet_service.cc:1468] Processing CreateTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 (DEFAULT_TABLE table=default.Table_c586a09ba29642f394d315d9b8ae758c [id=cc648249ae3c443e91004add091222de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.450968 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4bd19f9578f4fa19a6daef427e4a6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.451053 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a72c6040421642578fac307ea9914ef6 (DEFAULT_TABLE table=default.Table_b8bdb1307ac249ac89d252814049090f [id=6a41ce2e3dcf43ce9edb6392a244d9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.451382 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72c6040421642578fac307ea9914ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.452437 894 tablet_service.cc:1468] Processing CreateTablet for tablet a72c6040421642578fac307ea9914ef6 (DEFAULT_TABLE table=default.Table_b8bdb1307ac249ac89d252814049090f [id=6a41ce2e3dcf43ce9edb6392a244d9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.452684 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72c6040421642578fac307ea9914ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.453403 893 tablet_service.cc:1468] Processing CreateTablet for tablet d550a0ce24e24055947c8344c79bcca3 (DEFAULT_TABLE table=default.Table_daa3abe221c44551848996265764f670 [id=5a1fd538766d4b189cb5dd04bf0b2314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.453488 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 (DEFAULT_TABLE table=default.Table_a37f4562238b452abf51e6e3d547ca36 [id=c6a48bb9a6874c40b2f7679f2db9e756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.453648 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d550a0ce24e24055947c8344c79bcca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.453758 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431dc961cdcf4cd38b521d8d8fc6f406. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.456871 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d550a0ce24e24055947c8344c79bcca3 (DEFAULT_TABLE table=default.Table_daa3abe221c44551848996265764f670 [id=5a1fd538766d4b189cb5dd04bf0b2314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.457185 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d550a0ce24e24055947c8344c79bcca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.460045 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 (DEFAULT_TABLE table=default.Table_c586a09ba29642f394d315d9b8ae758c [id=cc648249ae3c443e91004add091222de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.460649 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4bd19f9578f4fa19a6daef427e4a6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.464645 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 0d46683318dc4c04a617a99f3e620734 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.469012 452 master_service.cc:948] ReplaceTablet: received request to replace tablet ee70cbe4d80e41b99234f1e1b001434b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.470511 892 tablet_service.cc:1468] Processing CreateTablet for tablet 4562bba09498449fbff9b7c60bad58fc (DEFAULT_TABLE table=default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.470628 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4562bba09498449fbff9b7c60bad58fc (DEFAULT_TABLE table=default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.470790 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4562bba09498449fbff9b7c60bad58fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.470897 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4562bba09498449fbff9b7c60bad58fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.475965 896 tablet_service.cc:1468] Processing CreateTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 (DEFAULT_TABLE table=default.Table_a37f4562238b452abf51e6e3d547ca36 [id=c6a48bb9a6874c40b2f7679f2db9e756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.476285 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431dc961cdcf4cd38b521d8d8fc6f406. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.476464 1158 ts_tablet_manager.cc:1918] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.476572 1158 log.cc:1199] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8453efdd649747a78c94eaa82fcde378
I20250629 01:51:39.476897 1158 ts_tablet_manager.cc:1939] T 8453efdd649747a78c94eaa82fcde378 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:39.478885 755 tablet_service.cc:1468] Processing CreateTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 (DEFAULT_TABLE table=default.Table_c586a09ba29642f394d315d9b8ae758c [id=cc648249ae3c443e91004add091222de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.479452 1156 raft_consensus.cc:2241] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:39.481703 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8453efdd649747a78c94eaa82fcde378 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]) successfully deleted
I20250629 01:51:39.482509 1154 tablet_bootstrap.cc:492] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.485199 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3963492a9da142409d4958813415dee6" } new_table_name: "default.Table_a50f761fd3bc4563ad4fbf38a6b97fd0" modify_external_catalogs: true
I20250629 01:51:39.489508 1154 ts_tablet_manager.cc:1397] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.004s
I20250629 01:51:39.489754 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu with pid 563
I20250629 01:51:39.490247 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 9af16f666fc6479ab67e449d1e962f8f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.490301 1154 raft_consensus.cc:357] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.490895 1159 tablet_bootstrap.cc:492] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.491006 1159 ts_tablet_manager.cc:1397] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.398s user 0.020s sys 0.087s
I20250629 01:51:39.490993 1154 raft_consensus.cc:383] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.491073 1154 raft_consensus.cc:738] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.491160 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4bd19f9578f4fa19a6daef427e4a6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.491362 1154 consensus_queue.cc:260] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.491734 1159 raft_consensus.cc:357] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.491866 1159 raft_consensus.cc:383] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.491937 1159 raft_consensus.cc:738] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.492149 1159 consensus_queue.cc:260] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.492848 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_036318a528c94f2fab8d77b241776f70" } modify_external_catalogs: true
I20250629 01:51:39.493010 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_036318a528c94f2fab8d77b241776f70" } modify_external_catalogs: true
I20250629 01:51:39.493674 1154 ts_tablet_manager.cc:1428] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:39.494076 1159 ts_tablet_manager.cc:1428] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:39.494300 1159 tablet_bootstrap.cc:492] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.495129 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ee70cbe4d80e41b99234f1e1b001434b
I20250629 01:51:39.495486 452 catalog_manager.cc:6652] ReplaceTablet: tablet ee70cbe4d80e41b99234f1e1b001434b deleted and replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0
I20250629 01:51:39.495604 1159 tablet_bootstrap.cc:654] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.496824 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ac26cba8a92945dd9acd10e8a76172ca
I20250629 01:51:39.498937 1154 tablet_bootstrap.cc:492] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.499661 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ac26cba8a92945dd9acd10e8a76172ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.500360 1154 tablet_bootstrap.cc:654] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.500540 896 tablet_service.cc:1515] Processing DeleteTablet for tablet ac26cba8a92945dd9acd10e8a76172ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.501055 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ee70cbe4d80e41b99234f1e1b001434b with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.502487 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ac26cba8a92945dd9acd10e8a76172ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac26cba8a92945dd9acd10e8a76172ca already in progress: creating tablet
W20250629 01:51:39.502692 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ee70cbe4d80e41b99234f1e1b001434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee70cbe4d80e41b99234f1e1b001434b already in progress: creating tablet
I20250629 01:51:39.503157 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 4562bba09498449fbff9b7c60bad58fc from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.503281 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 8453efdd649747a78c94eaa82fcde378 with delete_type TABLET_DATA_DELETED (replaced by tablet 1185e85345664dfd92c134c5552e7d48) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.503494 1157 tablet_replica.cc:331] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.503641 1157 raft_consensus.cc:2241] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.503777 1157 raft_consensus.cc:2270] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:39.504251 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ac26cba8a92945dd9acd10e8a76172ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac26cba8a92945dd9acd10e8a76172ca already in progress: creating tablet
I20250629 01:51:39.504274 1157 ts_tablet_manager.cc:1905] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.504477 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ac26cba8a92945dd9acd10e8a76172ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.505146 756 tablet_service.cc:1468] Processing CreateTablet for tablet a72c6040421642578fac307ea9914ef6 (DEFAULT_TABLE table=default.Table_b8bdb1307ac249ac89d252814049090f [id=6a41ce2e3dcf43ce9edb6392a244d9a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.507282 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet ee70cbe4d80e41b99234f1e1b001434b with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.507714 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ee70cbe4d80e41b99234f1e1b001434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee70cbe4d80e41b99234f1e1b001434b already in progress: creating tablet
I20250629 01:51:39.508015 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.508204 1158 tablet_replica.cc:331] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:39.508351 1158 raft_consensus.cc:2241] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.508486 1158 raft_consensus.cc:2270] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.509075 1158 ts_tablet_manager.cc:1905] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.509264 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b8bdb1307ac249ac89d252814049090f" } new_table_name: "default.Table_8eedc26a9ab14d5c8dd30631e6ec1fe8" modify_external_catalogs: true
I20250629 01:51:39.510106 1157 ts_tablet_manager.cc:1918] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.510208 1157 log.cc:1199] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8453efdd649747a78c94eaa82fcde378
I20250629 01:51:39.510483 1157 ts_tablet_manager.cc:1939] T 8453efdd649747a78c94eaa82fcde378 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.510818 752 tablet_service.cc:1468] Processing CreateTablet for tablet d550a0ce24e24055947c8344c79bcca3 (DEFAULT_TABLE table=default.Table_daa3abe221c44551848996265764f670 [id=5a1fd538766d4b189cb5dd04bf0b2314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.511224 1155 tablet_bootstrap.cc:492] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:39.511267 1188 consensus_queue.cc:1162] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250629 01:51:39.511515 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a72c6040421642578fac307ea9914ef6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.512198 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc6f34d6b31849e190ebfaf1a273d29c consensus state has no leader
I20250629 01:51:39.512853 1156 pending_rounds.cc:70] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:39.513792 753 tablet_service.cc:1468] Processing CreateTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 (DEFAULT_TABLE table=default.Table_a37f4562238b452abf51e6e3d547ca36 [id=c6a48bb9a6874c40b2f7679f2db9e756]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.514117 1156 pending_rounds.cc:77] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759139927994368 op_type: NO_OP noop_request { }
I20250629 01:51:39.513701 757 tablet_service.cc:1468] Processing CreateTablet for tablet 4562bba09498449fbff9b7c60bad58fc (DEFAULT_TABLE table=default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.514379 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_601ff9b6b68b4acb9206a64afa4339a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.514781 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_601ff9b6b68b4acb9206a64afa4339a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.515273 1154 tablet_bootstrap.cc:492] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.515383 1154 ts_tablet_manager.cc:1397] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20250629 01:51:39.515975 1154 raft_consensus.cc:357] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.516053 1159 tablet_bootstrap.cc:492] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.516156 1154 raft_consensus.cc:383] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.516193 1159 ts_tablet_manager.cc:1397] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250629 01:51:39.516242 1154 raft_consensus.cc:738] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.516436 1154 consensus_queue.cc:260] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.516826 1154 ts_tablet_manager.cc:1428] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:39.517108 1154 tablet_bootstrap.cc:492] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.517264 1159 raft_consensus.cc:357] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.517468 1159 raft_consensus.cc:383] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.517531 1159 raft_consensus.cc:738] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.517992 1155 tablet_bootstrap.cc:654] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.518046 1159 consensus_queue.cc:260] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.518489 1154 tablet_bootstrap.cc:654] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.518360 1156 raft_consensus.cc:2887] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:39.519119 1156 raft_consensus.cc:2270] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.519318 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431dc961cdcf4cd38b521d8d8fc6f406. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.519510 1159 ts_tablet_manager.cc:1428] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:39.519757 1159 tablet_bootstrap.cc:492] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.520210 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d550a0ce24e24055947c8344c79bcca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.521051 1159 tablet_bootstrap.cc:654] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.521739 1158 ts_tablet_manager.cc:1918] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.521857 1158 log.cc:1199] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/17dfc03d30ed40a498cae5efdc5445c7
I20250629 01:51:39.522166 1158 ts_tablet_manager.cc:1939] T 17dfc03d30ed40a498cae5efdc5445c7 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:39.522890 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8453efdd649747a78c94eaa82fcde378 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]) successfully deleted
I20250629 01:51:39.523520 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 17dfc03d30ed40a498cae5efdc5445c7 (table default.Table_4fac489880854041a135f94794583e93 [id=228c92a41618432cb3f0986698fc74c8]) successfully deleted
I20250629 01:51:39.522709 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.523962 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4562bba09498449fbff9b7c60bad58fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.525100 1155 tablet_bootstrap.cc:492] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.525244 1155 ts_tablet_manager.cc:1397] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.002s
I20250629 01:51:39.525604 751 tablet_service.cc:1515] Processing DeleteTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca with delete_type TABLET_DATA_DELETED (replaced by tablet d237b9f962024ecb967f86a5730a5039) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.526459 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cc260579c0da4ed38d9b0168e469c1b5
W20250629 01:51:39.527271 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a72c6040421642578fac307ea9914ef6 consensus state has no leader
I20250629 01:51:39.525978 1155 raft_consensus.cc:357] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.527941 1155 raft_consensus.cc:383] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.528106 894 tablet_service.cc:1515] Processing DeleteTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.529639 1155 raft_consensus.cc:738] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.529876 1159 tablet_bootstrap.cc:492] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.529984 1159 ts_tablet_manager.cc:1397] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250629 01:51:39.530371 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.530664 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 9af16f666fc6479ab67e449d1e962f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.530767 1159 raft_consensus.cc:357] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.530856 1157 tablet_replica.cc:331] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.530913 1159 raft_consensus.cc:383] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.530933 1154 tablet_bootstrap.cc:492] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.530984 1159 raft_consensus.cc:738] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.531042 1157 raft_consensus.cc:2241] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.531055 1154 ts_tablet_manager.cc:1397] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250629 01:51:39.531128 1156 ts_tablet_manager.cc:1905] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.531160 1157 raft_consensus.cc:2270] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.531713 1157 ts_tablet_manager.cc:1905] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.531668 1154 raft_consensus.cc:357] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.531739 1155 consensus_queue.cc:260] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.531827 1154 raft_consensus.cc:383] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.531880 1154 raft_consensus.cc:738] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.532056 1154 consensus_queue.cc:260] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.532438 1154 ts_tablet_manager.cc:1428] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:39.532694 1154 tablet_bootstrap.cc:492] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:39.533896 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cc260579c0da4ed38d9b0168e469c1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc260579c0da4ed38d9b0168e469c1b5 already in progress: creating tablet
I20250629 01:51:39.534726 1154 tablet_bootstrap.cc:654] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.535861 1158 tablet_replica.cc:331] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:39.535989 1158 raft_consensus.cc:2241] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.536135 1158 raft_consensus.cc:2270] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.536856 1158 ts_tablet_manager.cc:1905] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.537413 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b966a9c2749640808cf97059845d3b44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:38 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.537609 1157 ts_tablet_manager.cc:1918] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.537719 1157 log.cc:1199] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9af16f666fc6479ab67e449d1e962f8f
I20250629 01:51:39.538002 1157 ts_tablet_manager.cc:1939] T 9af16f666fc6479ab67e449d1e962f8f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.538072 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_66f7cc85f9864d11a6f9cb1d64bfa69d" } new_table_name: "default.Table_3194edfea46f4f9e8e989565b3438162" modify_external_catalogs: true
I20250629 01:51:39.531194 1159 consensus_queue.cc:260] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.539067 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6385bcc046a24e29b29108aa7a0e14d3" } new_table_name: "default.Table_8917f34339fb40e0a110ff85060ad961" modify_external_catalogs: true
I20250629 01:51:39.539445 1159 ts_tablet_manager.cc:1428] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250629 01:51:39.539705 1159 tablet_bootstrap.cc:492] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.540024 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5a356c61f0ef4ddfba46aac25d270004" } new_table_name: "default.Table_81df428605084d568b00885fe59d5e29" modify_external_catalogs: true
I20250629 01:51:39.541133 1159 tablet_bootstrap.cc:654] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.541427 1155 ts_tablet_manager.cc:1428] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:39.541749 1155 tablet_bootstrap.cc:492] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:39.544406 1154 tablet_bootstrap.cc:492] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.544546 1154 ts_tablet_manager.cc:1397] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20250629 01:51:39.545194 1154 raft_consensus.cc:357] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.545343 1154 raft_consensus.cc:383] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.545432 1154 raft_consensus.cc:738] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.545645 1154 consensus_queue.cc:260] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.546502 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0803b366c30642e681bff17044760ace" } modify_external_catalogs: true
I20250629 01:51:39.546687 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0803b366c30642e681bff17044760ace" } modify_external_catalogs: true
I20250629 01:51:39.547895 1154 ts_tablet_manager.cc:1428] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:39.547955 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9af16f666fc6479ab67e449d1e962f8f (table default.Table_e73a4b091c874ab0b2f5d9423f300110 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]) successfully deleted
I20250629 01:51:39.548724 894 tablet_service.cc:1468] Processing CreateTablet for tablet 0f40aefcb7e14612bf2d435dfa37d66e (DEFAULT_TABLE table=default.Table_5a356c61f0ef4ddfba46aac25d270004 [id=db4ab9642af04dd8b4fe207346a42384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.549162 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f40aefcb7e14612bf2d435dfa37d66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.549489 1154 tablet_bootstrap.cc:492] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.550559 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 0f40aefcb7e14612bf2d435dfa37d66e (DEFAULT_TABLE table=default.Table_5a356c61f0ef4ddfba46aac25d270004 [id=db4ab9642af04dd8b4fe207346a42384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.550881 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f40aefcb7e14612bf2d435dfa37d66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.551067 1154 tablet_bootstrap.cc:654] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.551584 1159 tablet_bootstrap.cc:492] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.551707 1159 ts_tablet_manager.cc:1397] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20250629 01:51:39.552304 1159 raft_consensus.cc:357] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.552474 1159 raft_consensus.cc:383] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.552543 1159 raft_consensus.cc:738] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.552769 1159 consensus_queue.cc:260] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.553825 1159 ts_tablet_manager.cc:1428] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:39.554906 1159 tablet_bootstrap.cc:492] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.556444 894 tablet_service.cc:1468] Processing CreateTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c (DEFAULT_TABLE table=default.Table_7a50e6b289ed401092480b4d315d2840 [id=c1e051380c514aa59fd3d4d84ee465d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.556800 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c28ea445cec4b35be800dbc8c37af7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.557021 890 tablet_service.cc:1468] Processing CreateTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 (DEFAULT_TABLE table=default.Table_e201257c73664abc93462558cfcdc261 [id=052b54452f124ba79d77e66ee2d68b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.557024 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_66f7cc85f9864d11a6f9cb1d64bfa69d [id=ac80888442774af495a53bba44dfa763]). Expected version 1 got 0
I20250629 01:51:39.557363 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bb791d4a38481aa05a4dbbe3577461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.556520 1159 tablet_bootstrap.cc:654] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.560633 751 tablet_service.cc:1468] Processing CreateTablet for tablet 0f40aefcb7e14612bf2d435dfa37d66e (DEFAULT_TABLE table=default.Table_5a356c61f0ef4ddfba46aac25d270004 [id=db4ab9642af04dd8b4fe207346a42384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.561100 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f40aefcb7e14612bf2d435dfa37d66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.561667 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_daa3abe221c44551848996265764f670" } modify_external_catalogs: true
I20250629 01:51:39.561826 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_daa3abe221c44551848996265764f670" } modify_external_catalogs: true
I20250629 01:51:39.563956 750 tablet_service.cc:1468] Processing CreateTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c (DEFAULT_TABLE table=default.Table_7a50e6b289ed401092480b4d315d2840 [id=c1e051380c514aa59fd3d4d84ee465d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.565055 446 master_service.cc:948] ReplaceTablet: received request to replace tablet d550a0ce24e24055947c8344c79bcca3 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.565833 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c28ea445cec4b35be800dbc8c37af7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.565863 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7724653350cb485cb25696194e364d3f" } modify_external_catalogs: true
I20250629 01:51:39.566004 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7724653350cb485cb25696194e364d3f" } modify_external_catalogs: true
I20250629 01:51:39.569715 894 tablet_service.cc:1515] Processing DeleteTablet for tablet f6690e00dfaa4d12be27e2fed9d835ca with delete_type TABLET_DATA_DELETED (replaced by tablet d237b9f962024ecb967f86a5730a5039) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.569947 1157 tablet_replica.cc:331] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.570108 1157 raft_consensus.cc:2241] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.570272 1157 raft_consensus.cc:2270] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.570573 1159 tablet_bootstrap.cc:492] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.570686 1159 ts_tablet_manager.cc:1397] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250629 01:51:39.574450 1159 raft_consensus.cc:357] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.574585 1157 ts_tablet_manager.cc:1905] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.574688 1159 raft_consensus.cc:383] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.574808 1159 raft_consensus.cc:738] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.575028 1159 consensus_queue.cc:260] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:39.575349 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88031f8d8cbc44e4b8c53d9cd5360552 consensus state has no leader
W20250629 01:51:39.575680 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57f7f71ab5e54d249141c89d9e806111 consensus state has no leader
W20250629 01:51:39.575953 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f40aefcb7e14612bf2d435dfa37d66e consensus state has no leader
I20250629 01:51:39.576228 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4b3395b9c2d04a6f8a227630087d85d8
I20250629 01:51:39.576556 1159 ts_tablet_manager.cc:1428] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:39.576797 1159 tablet_bootstrap.cc:492] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.577452 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:39.577586 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 17dfc03d30ed40a498cae5efdc5445c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 20958c5e88454e129aa8efefb63b7f14) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.577864 452 catalog_manager.cc:6652] ReplaceTablet: tablet 4562bba09498449fbff9b7c60bad58fc deleted and replaced by tablet 97e03c52aeb4432cab002abdae5109ba
I20250629 01:51:39.578469 1159 tablet_bootstrap.cc:654] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.579349 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.580381 1157 ts_tablet_manager.cc:1918] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.580487 1157 log.cc:1199] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f6690e00dfaa4d12be27e2fed9d835ca
I20250629 01:51:39.580492 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.580787 1157 ts_tablet_manager.cc:1939] T f6690e00dfaa4d12be27e2fed9d835ca P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.581000 754 tablet_service.cc:1515] Processing DeleteTablet for tablet ee70cbe4d80e41b99234f1e1b001434b with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.583958 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 (DEFAULT_TABLE table=default.Table_e201257c73664abc93462558cfcdc261 [id=052b54452f124ba79d77e66ee2d68b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.584853 894 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.585055 1157 tablet_replica.cc:331] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.585201 1157 raft_consensus.cc:2241] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.585286 1155 tablet_bootstrap.cc:654] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.585330 1157 raft_consensus.cc:2270] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.585913 1157 ts_tablet_manager.cc:1905] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.588560 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c (DEFAULT_TABLE table=default.Table_7a50e6b289ed401092480b4d315d2840 [id=c1e051380c514aa59fd3d4d84ee465d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.588752 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.588894 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c28ea445cec4b35be800dbc8c37af7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.589214 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.590610 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 95c14dacdd0d4156ba7b172edf255ae3
I20250629 01:51:39.590652 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d550a0ce24e24055947c8344c79bcca3
I20250629 01:51:39.591889 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f6690e00dfaa4d12be27e2fed9d835ca (table default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]) successfully deleted
I20250629 01:51:39.591974 755 tablet_service.cc:1468] Processing CreateTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 (DEFAULT_TABLE table=default.Table_e201257c73664abc93462558cfcdc261 [id=052b54452f124ba79d77e66ee2d68b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.592801 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.593200 894 tablet_service.cc:1515] Processing DeleteTablet for tablet d550a0ce24e24055947c8344c79bcca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.599057 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:39.599766 446 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:39.600489 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d550a0ce24e24055947c8344c79bcca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.600692 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.601261 754 tablet_service.cc:1515] Processing DeleteTablet for tablet d550a0ce24e24055947c8344c79bcca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.601763 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bb791d4a38481aa05a4dbbe3577461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.601864 1157 ts_tablet_manager.cc:1918] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.601986 1157 log.cc:1199] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/da27c51828684053b4f777739b0f2e51
I20250629 01:51:39.602325 1157 ts_tablet_manager.cc:1939] T da27c51828684053b4f777739b0f2e51 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.603785 894 tablet_service.cc:1468] Processing CreateTablet for tablet 78a5f38231a34efaab177a23cf693d51 (DEFAULT_TABLE table=default.Table_601ff9b6b68b4acb9206a64afa4339a8 [id=486534f71b544cee8fd56472d6806f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.604072 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a5f38231a34efaab177a23cf693d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.604023 754 tablet_service.cc:1468] Processing CreateTablet for tablet 78a5f38231a34efaab177a23cf693d51 (DEFAULT_TABLE table=default.Table_601ff9b6b68b4acb9206a64afa4339a8 [id=486534f71b544cee8fd56472d6806f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.604521 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a5f38231a34efaab177a23cf693d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.608600 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.608917 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae96f2da36744198e3d38cb07bb47a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.610878 1156 ts_tablet_manager.cc:1918] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:39.611228 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.612234 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2d68cf4935ca467fa49eb226d95d067a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.612682 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d68cf4935ca467fa49eb226d95d067a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.614773 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f749b62b51984bb6bdd8d3791a7bd03b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.614982 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet da27c51828684053b4f777739b0f2e51 (table default.Table_7017fd5ca5754ea0b74a91ffa9017025 [id=f4a6794121364ee19c24ef8ff99d8f6d]) successfully deleted
W20250629 01:51:39.615224 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f749b62b51984bb6bdd8d3791a7bd03b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.616164 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c071543aa01840ecaaa8d2af64bf15b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.616554 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c071543aa01840ecaaa8d2af64bf15b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.617766 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_71d29e8b5c32466eb1f9403fb88cfbba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.618134 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71d29e8b5c32466eb1f9403fb88cfbba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.621512 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7abb332bd3ed4d7299fc9f6959013250"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.621891 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7abb332bd3ed4d7299fc9f6959013250 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.622570 1156 log.cc:1199] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9af16f666fc6479ab67e449d1e962f8f
I20250629 01:51:39.613052 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.623073 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6f9bb7305f9d42fbb5b12690468aed5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.623445 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f9bb7305f9d42fbb5b12690468aed5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.623497 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae96f2da36744198e3d38cb07bb47a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.613255 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.601706 752 tablet_service.cc:1515] Processing DeleteTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.628574 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.628962 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 78a5f38231a34efaab177a23cf693d51 (DEFAULT_TABLE table=default.Table_601ff9b6b68b4acb9206a64afa4339a8 [id=486534f71b544cee8fd56472d6806f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.623096 1156 ts_tablet_manager.cc:1939] T 9af16f666fc6479ab67e449d1e962f8f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.629935 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9af16f666fc6479ab67e449d1e962f8f (table default.Table_e73a4b091c874ab0b2f5d9423f300110 [id=32ef01a6df3446b4b5ab3f1ef7e2bbf3]) successfully deleted
I20250629 01:51:39.630095 890 tablet_service.cc:1468] Processing CreateTablet for tablet 97e03c52aeb4432cab002abdae5109ba (DEFAULT_TABLE table=default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.630401 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e03c52aeb4432cab002abdae5109ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.633519 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 97e03c52aeb4432cab002abdae5109ba (DEFAULT_TABLE table=default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.633985 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:39.634742 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d237b9f962024ecb967f86a5730a5039 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d237b9f962024ecb967f86a5730a5039 already in progress: creating tablet
W20250629 01:51:39.635407 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 88fdf1f8bb96449ba472990092cfe8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88fdf1f8bb96449ba472990092cfe8cf already in progress: creating tablet
W20250629 01:51:39.637660 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7ab05d7564a8476c9f56c06687d11f36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ab05d7564a8476c9f56c06687d11f36 already in progress: creating tablet
W20250629 01:51:39.638235 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0d46683318dc4c04a617a99f3e620734 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d46683318dc4c04a617a99f3e620734 already in progress: creating tablet
W20250629 01:51:39.638819 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ac26cba8a92945dd9acd10e8a76172ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac26cba8a92945dd9acd10e8a76172ca already in progress: creating tablet
I20250629 01:51:39.639187 1156 tablet_replica.cc:331] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.639328 1156 raft_consensus.cc:2241] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.639468 1156 raft_consensus.cc:2270] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.640578 1156 ts_tablet_manager.cc:1905] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:39.641465 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4b3395b9c2d04a6f8a227630087d85d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b3395b9c2d04a6f8a227630087d85d8 already in progress: creating tablet
I20250629 01:51:39.633333 752 tablet_service.cc:1468] Processing CreateTablet for tablet 97e03c52aeb4432cab002abdae5109ba (DEFAULT_TABLE table=default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:39.641952 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4562bba09498449fbff9b7c60bad58fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4562bba09498449fbff9b7c60bad58fc already in progress: creating tablet
W20250629 01:51:39.642463 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 95c14dacdd0d4156ba7b172edf255ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c14dacdd0d4156ba7b172edf255ae3 already in progress: creating tablet
W20250629 01:51:39.642930 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d550a0ce24e24055947c8344c79bcca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d550a0ce24e24055947c8344c79bcca3 already in progress: creating tablet
I20250629 01:51:39.644039 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e03c52aeb4432cab002abdae5109ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.644349 1155 tablet_bootstrap.cc:492] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.644289 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f36b714b857b4ffdb47fc4eb3261e7a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.644523 1155 ts_tablet_manager.cc:1397] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.103s user 0.007s sys 0.001s
W20250629 01:51:39.644702 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f36b714b857b4ffdb47fc4eb3261e7a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.645303 1155 raft_consensus.cc:357] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.645498 1155 raft_consensus.cc:383] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.645570 1155 raft_consensus.cc:738] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.645861 1155 consensus_queue.cc:260] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.646992 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dd1bc966fc0543dc9370ae14d27d056b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.647392 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd1bc966fc0543dc9370ae14d27d056b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.647787 1155 ts_tablet_manager.cc:1428] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:39.648023 1155 tablet_bootstrap.cc:492] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:39.649058 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7bb791d4a38481aa05a4dbbe3577461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.649585 1155 tablet_bootstrap.cc:654] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.650132 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_654038c5004a437482989db6feea6573"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.650521 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_654038c5004a437482989db6feea6573 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.651201 1159 tablet_bootstrap.cc:492] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.651335 1159 ts_tablet_manager.cc:1397] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.018s
I20250629 01:51:39.651975 1159 raft_consensus.cc:357] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.652182 1159 raft_consensus.cc:383] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.652245 1159 raft_consensus.cc:738] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.652465 1159 consensus_queue.cc:260] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.653599 1159 ts_tablet_manager.cc:1428] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:39.653856 1159 tablet_bootstrap.cc:492] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.654325 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:39.655978 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 (DEFAULT_TABLE table=default.Table_71d29e8b5c32466eb1f9403fb88cfbba [id=ba643eccd0e54ce083459bf535db1fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.656870 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 (DEFAULT_TABLE table=default.Table_71d29e8b5c32466eb1f9403fb88cfbba [id=ba643eccd0e54ce083459bf535db1fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.657675 755 tablet_service.cc:1468] Processing CreateTablet for tablet d1956051272e450ab2ba3322ba23ab69 (DEFAULT_TABLE table=default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.658252 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1956051272e450ab2ba3322ba23ab69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.658929 754 tablet_service.cc:1468] Processing CreateTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 (DEFAULT_TABLE table=default.Table_f749b62b51984bb6bdd8d3791a7bd03b [id=9a2d24af85fb40b3a4c49f37cca3ab62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.659344 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.660459 750 tablet_service.cc:1515] Processing DeleteTablet for tablet da27c51828684053b4f777739b0f2e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.660580 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b1ecf098ab4e498391e79cfb1c60b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.663221 1159 tablet_bootstrap.cc:654] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.665015 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 (DEFAULT_TABLE table=default.Table_71d29e8b5c32466eb1f9403fb88cfbba [id=ba643eccd0e54ce083459bf535db1fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.665344 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.668977 891 tablet_service.cc:1468] Processing CreateTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 (DEFAULT_TABLE table=default.Table_f749b62b51984bb6bdd8d3791a7bd03b [id=9a2d24af85fb40b3a4c49f37cca3ab62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.669299 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b1ecf098ab4e498391e79cfb1c60b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.669449 1032 tablet_service.cc:1468] Processing CreateTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 (DEFAULT_TABLE table=default.Table_f749b62b51984bb6bdd8d3791a7bd03b [id=9a2d24af85fb40b3a4c49f37cca3ab62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.671232 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a22af67c80541b38f2d4c121fdeb6e4 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.671803 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59bcc265cd41a7852b98dc84b34aff with delete_type TABLET_DATA_DELETED (replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.673213 894 tablet_service.cc:1468] Processing CreateTablet for tablet d1956051272e450ab2ba3322ba23ab69 (DEFAULT_TABLE table=default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.675473 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1956051272e450ab2ba3322ba23ab69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.677888 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a5f38231a34efaab177a23cf693d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.678179 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.678222 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d1956051272e450ab2ba3322ba23ab69 (DEFAULT_TABLE table=default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.678365 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6b1ecf098ab4e498391e79cfb1c60b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.678373 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 990d1fcdac1c406facb2da722421d7f4 (DEFAULT_TABLE table=default.Table_c071543aa01840ecaaa8d2af64bf15b7 [id=d02cba06b5bc499280a223f49faa9e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.679060 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1956051272e450ab2ba3322ba23ab69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.679400 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 990d1fcdac1c406facb2da722421d7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.681115 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4a22af67c80541b38f2d4c121fdeb6e4
I20250629 01:51:39.681474 453 catalog_manager.cc:6652] ReplaceTablet: tablet 4a22af67c80541b38f2d4c121fdeb6e4 deleted and replaced by tablet 5b4701123dfa467ebb6c95f9742b3874
I20250629 01:51:39.682015 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ae96f2da36744198e3d38cb07bb47a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.682044 1156 ts_tablet_manager.cc:1918] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.682137 1156 log.cc:1199] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f6690e00dfaa4d12be27e2fed9d835ca
I20250629 01:51:39.682437 1156 ts_tablet_manager.cc:1939] T f6690e00dfaa4d12be27e2fed9d835ca P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.682672 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97e03c52aeb4432cab002abdae5109ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.683157 1156 tablet_replica.cc:331] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.683349 1156 raft_consensus.cc:2241] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:39.683470 964 tablet.cc:2378] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250629 01:51:39.683521 1156 raft_consensus.cc:2270] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.685415 1174 raft_consensus.cc:491] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:39.685556 1174 raft_consensus.cc:513] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.686466 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b4701123dfa467ebb6c95f9742b3874) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.689534 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b4701123dfa467ebb6c95f9742b3874) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.689626 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f6690e00dfaa4d12be27e2fed9d835ca (table default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]) successfully deleted
I20250629 01:51:39.690302 755 tablet_service.cc:1468] Processing CreateTablet for tablet 990d1fcdac1c406facb2da722421d7f4 (DEFAULT_TABLE table=default.Table_c071543aa01840ecaaa8d2af64bf15b7 [id=d02cba06b5bc499280a223f49faa9e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.690623 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 990d1fcdac1c406facb2da722421d7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.692409 1155 tablet_bootstrap.cc:492] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.692545 1155 ts_tablet_manager.cc:1397] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.003s
I20250629 01:51:39.693463 1155 raft_consensus.cc:357] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.693642 1155 raft_consensus.cc:383] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.693732 1155 raft_consensus.cc:738] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.693984 1155 consensus_queue.cc:260] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:39.695384 710 connection.cc:537] client connection to 127.0.104.60:32789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250629 01:51:39.695966 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
W20250629 01:51:39.696174 822 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:32789 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250629 01:51:39.697001 1174 leader_election.cc:290] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:39.697510 1173 raft_consensus.cc:491] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:39.697712 1173 raft_consensus.cc:513] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:39.698139 1155 ts_tablet_manager.cc:1428] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250629 01:51:39.698388 1155 tablet_bootstrap.cc:492] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:39.699293 1173 leader_election.cc:290] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:39.699916 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8c79e92b6cfe429088a6b6662f46412c" } modify_external_catalogs: true
I20250629 01:51:39.700059 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8c79e92b6cfe429088a6b6662f46412c" } modify_external_catalogs: true
I20250629 01:51:39.702067 1155 tablet_bootstrap.cc:654] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.703014 754 tablet_service.cc:1468] Processing CreateTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 (DEFAULT_TABLE table=default.Table_dd1bc966fc0543dc9370ae14d27d056b [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.703145 750 tablet_service.cc:1468] Processing CreateTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 (DEFAULT_TABLE table=default.Table_654038c5004a437482989db6feea6573 [id=0c84d077516244288b296db0048f3497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.703440 752 tablet_service.cc:1468] Processing CreateTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 (DEFAULT_TABLE table=default.Table_6f9bb7305f9d42fbb5b12690468aed5f [id=81e46758eee54c7caae081f9232e9071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.703799 753 tablet_service.cc:1468] Processing CreateTablet for tablet dd6013bb40c248f29f40eab2c3008f6b (DEFAULT_TABLE table=default.Table_7abb332bd3ed4d7299fc9f6959013250 [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.703505 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a229b4f9d19147a5990f799d1dc0e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.704262 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64477468c6da4b4ba282d8530eb5cbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.704685 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 267d9a050fa2414d87a04ba3c91dea03
I20250629 01:51:39.706787 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5a1e71ab9a84caea5c917d422336bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.708221 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6013bb40c248f29f40eab2c3008f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.713394 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71d29e8b5c32466eb1f9403fb88cfbba" } new_table_name: "default.Table_0b441ce248f2448eabf2d7e3cddff976" modify_external_catalogs: true
W20250629 01:51:39.716063 1209 negotiation.cc:337] Failed RPC negotiation. Trace:
0629 01:51:39.702314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:32789 (local address 127.0.0.1:56148)
0629 01:51:39.715046 (+ 12732us) negotiation.cc:107] Waiting for socket to connect
0629 01:51:39.715166 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":12630,"threads_started":1}
W20250629 01:51:39.716827 440 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer f304077528b34c61857a659e5873631d (127.0.104.60:32789): Couldn't send request to peer f304077528b34c61857a659e5873631d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:39.716917 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:39.717873 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 consensus state has no leader
I20250629 01:51:39.718086 1158 ts_tablet_manager.cc:1918] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.718218 1158 log.cc:1199] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/da27c51828684053b4f777739b0f2e51
I20250629 01:51:39.718577 754 tablet_service.cc:1468] Processing CreateTablet for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (DEFAULT_TABLE table=default.Table_f36b714b857b4ffdb47fc4eb3261e7a1 [id=08450a0d665043f8a1513dbb72d9edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.718921 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.722862 1157 tablet_replica.cc:331] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.723932 1156 ts_tablet_manager.cc:1905] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.727538 893 tablet_service.cc:1468] Processing CreateTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 (DEFAULT_TABLE table=default.Table_dd1bc966fc0543dc9370ae14d27d056b [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.728017 896 tablet_service.cc:1468] Processing CreateTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 (DEFAULT_TABLE table=default.Table_6f9bb7305f9d42fbb5b12690468aed5f [id=81e46758eee54c7caae081f9232e9071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.728343 892 tablet_service.cc:1468] Processing CreateTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 (DEFAULT_TABLE table=default.Table_654038c5004a437482989db6feea6573 [id=0c84d077516244288b296db0048f3497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.728936 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4fac489880854041a135f94794583e93" } new_table_name: "default.Table_b60d57f2a4b34062a9ed95ffd636f209" modify_external_catalogs: true
I20250629 01:51:39.730041 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a37f4562238b452abf51e6e3d547ca36" } modify_external_catalogs: true
I20250629 01:51:39.730216 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a37f4562238b452abf51e6e3d547ca36" } modify_external_catalogs: true
I20250629 01:51:39.731014 1157 raft_consensus.cc:2241] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.733968 752 tablet_service.cc:1468] Processing CreateTablet for tablet 5b4701123dfa467ebb6c95f9742b3874 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.734249 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4701123dfa467ebb6c95f9742b3874. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.734668 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fbbdafc99d2e477b92c7724d35c4f3ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.735138 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbbdafc99d2e477b92c7724d35c4f3ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.739642 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 20958c5e88454e129aa8efefb63b7f14 consensus state has no leader
I20250629 01:51:39.740646 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 431dc961cdcf4cd38b521d8d8fc6f406
I20250629 01:51:39.743129 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b4701123dfa467ebb6c95f9742b3874) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.743323 889 tablet_service.cc:1468] Processing CreateTablet for tablet dd6013bb40c248f29f40eab2c3008f6b (DEFAULT_TABLE table=default.Table_7abb332bd3ed4d7299fc9f6959013250 [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.743837 888 tablet_service.cc:1468] Processing CreateTablet for tablet 5b4701123dfa467ebb6c95f9742b3874 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.744426 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (DEFAULT_TABLE table=default.Table_f36b714b857b4ffdb47fc4eb3261e7a1 [id=08450a0d665043f8a1513dbb72d9edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.744879 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.745118 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5a1e71ab9a84caea5c917d422336bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.745592 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.748147 1157 raft_consensus.cc:2270] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.748831 1157 ts_tablet_manager.cc:1905] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.748904 1156 ts_tablet_manager.cc:1918] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.749011 1156 log.cc:1199] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/17dfc03d30ed40a498cae5efdc5445c7
I20250629 01:51:39.749307 1156 ts_tablet_manager.cc:1939] T 17dfc03d30ed40a498cae5efdc5445c7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.751681 1155 tablet_bootstrap.cc:492] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.751807 1155 ts_tablet_manager.cc:1397] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.005s
I20250629 01:51:39.752697 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6013bb40c248f29f40eab2c3008f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.754069 1155 raft_consensus.cc:357] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.754722 1156 tablet_replica.cc:331] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:39.754765 1208 negotiation.cc:337] Failed RPC negotiation. Trace:
0629 01:51:39.697801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:32789 (local address 127.0.104.1:59457)
0629 01:51:39.751055 (+ 53254us) negotiation.cc:107] Waiting for socket to connect
0629 01:51:39.751229 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":52326,"spinlock_wait_cycles":73984,"thread_start_us":52315,"threads_started":1}
I20250629 01:51:39.754890 1156 raft_consensus.cc:2241] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.755033 1156 raft_consensus.cc:2270] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.755935 1156 ts_tablet_manager.cc:1905] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.756075 1155 raft_consensus.cc:383] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.756194 1155 raft_consensus.cc:738] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.756448 1155 consensus_queue.cc:260] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.756632 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 17dfc03d30ed40a498cae5efdc5445c7 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]) successfully deleted
W20250629 01:51:39.756840 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4b3395b9c2d04a6f8a227630087d85d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b3395b9c2d04a6f8a227630087d85d8 already in progress: creating tablet
W20250629 01:51:39.757088 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4562bba09498449fbff9b7c60bad58fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4562bba09498449fbff9b7c60bad58fc already in progress: creating tablet
W20250629 01:51:39.757259 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ee70cbe4d80e41b99234f1e1b001434b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee70cbe4d80e41b99234f1e1b001434b already in progress: creating tablet
W20250629 01:51:39.757403 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d550a0ce24e24055947c8344c79bcca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d550a0ce24e24055947c8344c79bcca3 already in progress: creating tablet
W20250629 01:51:39.757678 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 95c14dacdd0d4156ba7b172edf255ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c14dacdd0d4156ba7b172edf255ae3 already in progress: creating tablet
W20250629 01:51:39.757860 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet cc260579c0da4ed38d9b0168e469c1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc260579c0da4ed38d9b0168e469c1b5 already in progress: creating tablet
I20250629 01:51:39.758705 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.759450 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4701123dfa467ebb6c95f9742b3874. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.760069 1155 ts_tablet_manager.cc:1428] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250629 01:51:39.760294 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64477468c6da4b4ba282d8530eb5cbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.761176 1155 tablet_bootstrap.cc:492] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:39.762677 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a229b4f9d19147a5990f799d1dc0e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.762689 1155 tablet_bootstrap.cc:654] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.766906 1156 ts_tablet_manager.cc:1918] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.767004 1156 log.cc:1199] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/da27c51828684053b4f777739b0f2e51
I20250629 01:51:39.767352 1156 ts_tablet_manager.cc:1939] T da27c51828684053b4f777739b0f2e51 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.768491 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 7ae96f2da36744198e3d38cb07bb47a0 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.771808 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet da27c51828684053b4f777739b0f2e51 (table default.Table_7017fd5ca5754ea0b74a91ffa9017025 [id=f4a6794121364ee19c24ef8ff99d8f6d]) successfully deleted
W20250629 01:51:39.771989 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4a22af67c80541b38f2d4c121fdeb6e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a22af67c80541b38f2d4c121fdeb6e4 already in progress: creating tablet
I20250629 01:51:39.774752 454 master_service.cc:948] ReplaceTablet: received request to replace tablet ac26cba8a92945dd9acd10e8a76172ca from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.785687 1155 tablet_bootstrap.cc:492] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.785840 1155 ts_tablet_manager.cc:1397] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.008s
I20250629 01:51:39.786736 1155 raft_consensus.cc:357] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.786928 1155 raft_consensus.cc:383] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.787039 1155 raft_consensus.cc:738] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.787320 1155 consensus_queue.cc:260] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.787771 1155 ts_tablet_manager.cc:1428] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:39.787985 1155 tablet_bootstrap.cc:492] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:39.797866 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 267d9a050fa2414d87a04ba3c91dea03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 267d9a050fa2414d87a04ba3c91dea03 already in progress: creating tablet
W20250629 01:51:39.798049 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 431dc961cdcf4cd38b521d8d8fc6f406 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 431dc961cdcf4cd38b521d8d8fc6f406 already in progress: creating tablet
I20250629 01:51:39.801265 1157 ts_tablet_manager.cc:1918] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.801443 1157 log.cc:1199] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/ab59bcc265cd41a7852b98dc84b34aff
I20250629 01:51:39.801887 1157 ts_tablet_manager.cc:1939] T ab59bcc265cd41a7852b98dc84b34aff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.802681 1155 tablet_bootstrap.cc:654] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.807591 1155 tablet_bootstrap.cc:492] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:39.807749 1155 ts_tablet_manager.cc:1397] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250629 01:51:39.808506 1155 raft_consensus.cc:357] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.808707 1155 raft_consensus.cc:383] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.808781 1155 raft_consensus.cc:738] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.809052 1155 consensus_queue.cc:260] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.810354 894 tablet_service.cc:1468] Processing CreateTablet for tablet 990d1fcdac1c406facb2da722421d7f4 (DEFAULT_TABLE table=default.Table_c071543aa01840ecaaa8d2af64bf15b7 [id=d02cba06b5bc499280a223f49faa9e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.811012 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 990d1fcdac1c406facb2da722421d7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.812170 1155 ts_tablet_manager.cc:1428] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:39.812709 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 (DEFAULT_TABLE table=default.Table_654038c5004a437482989db6feea6573 [id=0c84d077516244288b296db0048f3497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.813258 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a229b4f9d19147a5990f799d1dc0e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.814510 1155 tablet_bootstrap.cc:492] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:39.815878 1155 tablet_bootstrap.cc:654] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.817859 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 (DEFAULT_TABLE table=default.Table_6f9bb7305f9d42fbb5b12690468aed5f [id=81e46758eee54c7caae081f9232e9071]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.818243 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64477468c6da4b4ba282d8530eb5cbb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.819954 1027 tablet_service.cc:1468] Processing CreateTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 (DEFAULT_TABLE table=default.Table_dd1bc966fc0543dc9370ae14d27d056b [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.820641 1158 ts_tablet_manager.cc:1939] T da27c51828684053b4f777739b0f2e51 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:39.820742 1159 tablet_bootstrap.cc:492] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.820840 1159 ts_tablet_manager.cc:1397] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.167s user 0.017s sys 0.033s
W20250629 01:51:39.821413 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_8eedc26a9ab14d5c8dd30631e6ec1fe8 [id=6a41ce2e3dcf43ce9edb6392a244d9a2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a72c6040421642578fac307ea9914ef6 consensus state has no leader
I20250629 01:51:39.821521 1159 raft_consensus.cc:357] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.821712 1159 raft_consensus.cc:383] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.821794 1159 raft_consensus.cc:738] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.821969 1159 consensus_queue.cc:260] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.822360 1159 ts_tablet_manager.cc:1428] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:39.822698 1159 tablet_bootstrap.cc:492] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:39.823175 1154 tablet_bootstrap.cc:492] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.824275 1154 ts_tablet_manager.cc:1397] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.274s user 0.016s sys 0.026s
I20250629 01:51:39.824921 1159 tablet_bootstrap.cc:654] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.826910 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5a1e71ab9a84caea5c917d422336bc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.831681 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 5b4701123dfa467ebb6c95f9742b3874 (DEFAULT_TABLE table=default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.833774 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.835711 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:39.836023 1158 tablet_replica.cc:331] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:39.836479 446 master_service.cc:948] ReplaceTablet: received request to replace tablet d1956051272e450ab2ba3322ba23ab69 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.838912 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f749b62b51984bb6bdd8d3791a7bd03b" } new_table_name: "default.Table_7b25659fe37948109f17e9e6b15518cc" modify_external_catalogs: true
I20250629 01:51:39.839121 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet da27c51828684053b4f777739b0f2e51 (table default.Table_7017fd5ca5754ea0b74a91ffa9017025 [id=f4a6794121364ee19c24ef8ff99d8f6d]) successfully deleted
W20250629 01:51:39.839416 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet cc260579c0da4ed38d9b0168e469c1b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc260579c0da4ed38d9b0168e469c1b5 already in progress: creating tablet
I20250629 01:51:39.840128 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dd1bc966fc0543dc9370ae14d27d056b" } new_table_name: "default.Table_e2a173c972234867a7870c52678fc4bf" modify_external_catalogs: true
I20250629 01:51:39.841208 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e201257c73664abc93462558cfcdc261" } new_table_name: "default.Table_654a8089dcba4f64926b8b405150ffa5" modify_external_catalogs: true
I20250629 01:51:39.841686 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4701123dfa467ebb6c95f9742b3874. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.842195 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7a50e6b289ed401092480b4d315d2840" } modify_external_catalogs: true
I20250629 01:51:39.842320 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7a50e6b289ed401092480b4d315d2840" } modify_external_catalogs: true
I20250629 01:51:39.842532 1158 raft_consensus.cc:2241] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.842564 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0b441ce248f2448eabf2d7e3cddff976" } modify_external_catalogs: true
I20250629 01:51:39.842650 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0b441ce248f2448eabf2d7e3cddff976" } modify_external_catalogs: true
I20250629 01:51:39.842717 1158 raft_consensus.cc:2270] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.845228 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59bcc265cd41a7852b98dc84b34aff with delete_type TABLET_DATA_DELETED (replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.845265 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a22af67c80541b38f2d4c121fdeb6e4 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.845463 1156 tablet_replica.cc:331] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.845495 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b710052005c4a2aaade1dc60409aef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.845669 1156 raft_consensus.cc:2241] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.845830 1156 raft_consensus.cc:2270] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.846004 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.846493 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3ac2dc2474da47f7ba25608e622e9a68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.846632 1156 ts_tablet_manager.cc:1905] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:39.847021 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ac2dc2474da47f7ba25608e622e9a68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.847929 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.848296 1032 tablet_service.cc:1468] Processing CreateTablet for tablet dd6013bb40c248f29f40eab2c3008f6b (DEFAULT_TABLE table=default.Table_7abb332bd3ed4d7299fc9f6959013250 [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.839097 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f9bb7305f9d42fbb5b12690468aed5f" } modify_external_catalogs: true
I20250629 01:51:39.848950 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f9bb7305f9d42fbb5b12690468aed5f" } modify_external_catalogs: true
I20250629 01:51:39.852577 1156 ts_tablet_manager.cc:1918] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.852689 1156 log.cc:1199] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ab59bcc265cd41a7852b98dc84b34aff
I20250629 01:51:39.852964 1156 ts_tablet_manager.cc:1939] T ab59bcc265cd41a7852b98dc84b34aff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.853361 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6013bb40c248f29f40eab2c3008f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:39.853895 1156 tablet_replica.cc:331] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.854025 1156 raft_consensus.cc:2241] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.854161 1156 raft_consensus.cc:2270] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.854702 1156 ts_tablet_manager.cc:1905] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.858008 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_76c73b5d51504e688e3d59a43f6ab0a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.858541 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76c73b5d51504e688e3d59a43f6ab0a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.858561 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ab59bcc265cd41a7852b98dc84b34aff (table default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]) successfully deleted
I20250629 01:51:39.859994 1156 ts_tablet_manager.cc:1918] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.860112 1156 log.cc:1199] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0b710052005c4a2aaade1dc60409aef3
I20250629 01:51:39.860404 1156 ts_tablet_manager.cc:1939] T 0b710052005c4a2aaade1dc60409aef3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.860967 1158 ts_tablet_manager.cc:1905] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.861171 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0b710052005c4a2aaade1dc60409aef3 (table default.Table_c440da96423948c4bc188695d63152cc [id=64a5b540bb01488c92bb18807a03e2b7]) successfully deleted
I20250629 01:51:39.864401 1156 tablet_replica.cc:331] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:39.864559 1156 raft_consensus.cc:2241] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.864691 1156 raft_consensus.cc:2270] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.864950 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d1956051272e450ab2ba3322ba23ab69
I20250629 01:51:39.865417 446 catalog_manager.cc:6652] ReplaceTablet: tablet d1956051272e450ab2ba3322ba23ab69 deleted and replaced by tablet 02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:39.865736 752 tablet_service.cc:1515] Processing DeleteTablet for tablet d1956051272e450ab2ba3322ba23ab69 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d072bf72044dc48d8090660ecb7d47) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.866086 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d1956051272e450ab2ba3322ba23ab69 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d072bf72044dc48d8090660ecb7d47) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.866978 1156 ts_tablet_manager.cc:1905] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.867441 1158 ts_tablet_manager.cc:1918] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.867568 1158 log.cc:1199] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b966a9c2749640808cf97059845d3b44
I20250629 01:51:39.868077 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.868563 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.857338 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (DEFAULT_TABLE table=default.Table_f36b714b857b4ffdb47fc4eb3261e7a1 [id=08450a0d665043f8a1513dbb72d9edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:39.872849 1154 raft_consensus.cc:357] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:39.873811 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6b1ecf098ab4e498391e79cfb1c60b0 consensus state has no leader
I20250629 01:51:39.877096 1154 raft_consensus.cc:383] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.877215 1154 raft_consensus.cc:738] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.877972 1156 ts_tablet_manager.cc:1918] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.877956 1154 consensus_queue.cc:260] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.877982 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_96f437ef98bb46f69002f7508991afbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.878469 1154 ts_tablet_manager.cc:1428] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
W20250629 01:51:39.878497 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96f437ef98bb46f69002f7508991afbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.878948 1156 log.cc:1199] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/7ab05d7564a8476c9f56c06687d11f36
I20250629 01:51:39.879436 1156 ts_tablet_manager.cc:1939] T 7ab05d7564a8476c9f56c06687d11f36 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:39.880570 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 7ab05d7564a8476c9f56c06687d11f36 (table default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
W20250629 01:51:39.881681 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d1956051272e450ab2ba3322ba23ab69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1956051272e450ab2ba3322ba23ab69 already in progress: creating tablet
I20250629 01:51:39.884513 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7ae96f2da36744198e3d38cb07bb47a0
I20250629 01:51:39.884889 453 catalog_manager.cc:6652] ReplaceTablet: tablet 7ae96f2da36744198e3d38cb07bb47a0 deleted and replaced by tablet bd5b835ff8624ebf8956eeb965ce5950
I20250629 01:51:39.886420 1158 ts_tablet_manager.cc:1939] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:39.887120 1154 tablet_bootstrap.cc:492] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.889420 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5b835ff8624ebf8956eeb965ce5950) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:39.889657 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5b835ff8624ebf8956eeb965ce5950) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:39.891160 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:39.892136 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7ae96f2da36744198e3d38cb07bb47a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ae96f2da36744198e3d38cb07bb47a0 already in progress: creating tablet
I20250629 01:51:39.892336 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b966a9c2749640808cf97059845d3b44 (table default.Table_ca6aba8125c9496d86aa4dff2eed2591 [id=d4e47fc75da249138358461212d80e11]) successfully deleted
I20250629 01:51:39.895591 1144 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:39.895807 1144 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:51:39.896294 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4ff2476a31de4c299bf428196cd197ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.896754 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ff2476a31de4c299bf428196cd197ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.898939 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d550a0ce24e24055947c8344c79bcca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d550a0ce24e24055947c8344c79bcca3 already in progress: creating tablet
W20250629 01:51:39.902499 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 95c14dacdd0d4156ba7b172edf255ae3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95c14dacdd0d4156ba7b172edf255ae3 already in progress: creating tablet
W20250629 01:51:39.907020 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4562bba09498449fbff9b7c60bad58fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4562bba09498449fbff9b7c60bad58fc already in progress: creating tablet
W20250629 01:51:39.909936 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5a1e71ab9a84caea5c917d422336bc8 consensus state has no leader
I20250629 01:51:39.913010 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e4165d1f8e364693a7dd96d2e5e47fa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.913525 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.915256 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4b3395b9c2d04a6f8a227630087d85d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b3395b9c2d04a6f8a227630087d85d8 already in progress: creating tablet
W20250629 01:51:39.915879 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4a22af67c80541b38f2d4c121fdeb6e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a22af67c80541b38f2d4c121fdeb6e4 already in progress: creating tablet
W20250629 01:51:39.916551 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 267d9a050fa2414d87a04ba3c91dea03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 267d9a050fa2414d87a04ba3c91dea03 already in progress: creating tablet
W20250629 01:51:39.917145 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 431dc961cdcf4cd38b521d8d8fc6f406 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 431dc961cdcf4cd38b521d8d8fc6f406 already in progress: creating tablet
W20250629 01:51:39.917687 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d1956051272e450ab2ba3322ba23ab69 because the tablet was not found. No further retry: Not found: Tablet not found: d1956051272e450ab2ba3322ba23ab69
W20250629 01:51:39.918550 850 connection.cc:537] client connection to 127.0.104.60:32789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250629 01:51:39.918992 961 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:32789 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250629 01:51:39.920209 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet ab59bcc265cd41a7852b98dc84b34aff (table default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]) successfully deleted
W20250629 01:51:39.920585 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4a22af67c80541b38f2d4c121fdeb6e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a22af67c80541b38f2d4c121fdeb6e4 already in progress: creating tablet
W20250629 01:51:39.921787 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 267d9a050fa2414d87a04ba3c91dea03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 267d9a050fa2414d87a04ba3c91dea03 already in progress: creating tablet
W20250629 01:51:39.924535 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7bb791d4a38481aa05a4dbbe3577461 consensus state has no leader
I20250629 01:51:39.925462 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.925874 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4559cd80e6874b7fb3e1ea565b90fd03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.926322 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.926347 894 tablet_service.cc:1515] Processing DeleteTablet for tablet d1956051272e450ab2ba3322ba23ab69 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d072bf72044dc48d8090660ecb7d47) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.926967 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5b835ff8624ebf8956eeb965ce5950) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.927256 1154 tablet_bootstrap.cc:654] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.927528 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:39.927697 1157 tablet_replica.cc:331] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:39.927845 1157 raft_consensus.cc:2241] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:39.927983 1157 raft_consensus.cc:2270] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:39.928653 1157 ts_tablet_manager.cc:1905] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:39.934235 1157 ts_tablet_manager.cc:1918] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:39.934367 1157 log.cc:1199] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/7ab05d7564a8476c9f56c06687d11f36
I20250629 01:51:39.934717 1157 ts_tablet_manager.cc:1939] T 7ab05d7564a8476c9f56c06687d11f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:39.944756 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0c28ea445cec4b35be800dbc8c37af7c
I20250629 01:51:39.945359 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_30a1da646c914cde85078e3e1270a01e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:39.945636 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:39.945749 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30a1da646c914cde85078e3e1270a01e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:39.946269 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0c28ea445cec4b35be800dbc8c37af7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c28ea445cec4b35be800dbc8c37af7c already in progress: creating tablet
I20250629 01:51:39.944756 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6e7aa21adf4540bcb0f4c7f4e711f90b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:39.946987 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e7aa21adf4540bcb0f4c7f4e711f90b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:39.948491 449 master_service.cc:948] ReplaceTablet: received request to replace tablet d550a0ce24e24055947c8344c79bcca3 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:39.950052 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_601ff9b6b68b4acb9206a64afa4339a8" } modify_external_catalogs: true
I20250629 01:51:39.950219 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_601ff9b6b68b4acb9206a64afa4339a8" } modify_external_catalogs: true
I20250629 01:51:39.951596 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a50f761fd3bc4563ad4fbf38a6b97fd0" } new_table_name: "default.Table_c208a66223c24a10abbc5478f7469f04" modify_external_catalogs: true
I20250629 01:51:39.951712 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fef90accef4548ee9603f9b54d9a8a84" } new_table_name: "default.Table_84a213599d1e4030bd7ec42e4bb8c128" modify_external_catalogs: true
W20250629 01:51:39.957978 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 431dc961cdcf4cd38b521d8d8fc6f406 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 431dc961cdcf4cd38b521d8d8fc6f406 already in progress: creating tablet
I20250629 01:51:39.958421 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:39.958457 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d1956051272e450ab2ba3322ba23ab69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1956051272e450ab2ba3322ba23ab69 already in progress: creating tablet
W20250629 01:51:39.958846 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7ae96f2da36744198e3d38cb07bb47a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ae96f2da36744198e3d38cb07bb47a0 already in progress: creating tablet
I20250629 01:51:39.959232 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 7ab05d7564a8476c9f56c06687d11f36 (table default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
I20250629 01:51:39.960654 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:39.961090 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0c28ea445cec4b35be800dbc8c37af7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c28ea445cec4b35be800dbc8c37af7c already in progress: creating tablet
W20250629 01:51:39.963636 988 leader_election.cc:343] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: b966a9c2749640808cf97059845d3b44
W20250629 01:51:39.985587 986 connection.cc:537] client connection to 127.0.104.60:32789 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250629 01:51:39.993105 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7ae96f2da36744198e3d38cb07bb47a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ae96f2da36744198e3d38cb07bb47a0 already in progress: creating tablet
I20250629 01:51:39.987818 1154 tablet_bootstrap.cc:492] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:39.993793 1154 ts_tablet_manager.cc:1397] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.114s user 0.001s sys 0.003s
W20250629 01:51:39.994036 1097 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:32789 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250629 01:51:39.994551 1154 raft_consensus.cc:357] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.994730 1154 raft_consensus.cc:383] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.994817 1154 raft_consensus.cc:738] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.995031 1154 consensus_queue.cc:260] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.995466 1154 ts_tablet_manager.cc:1428] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250629 01:51:39.995484 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0c28ea445cec4b35be800dbc8c37af7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c28ea445cec4b35be800dbc8c37af7c already in progress: creating tablet
I20250629 01:51:39.995726 1154 tablet_bootstrap.cc:492] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:39.997268 1154 tablet_bootstrap.cc:654] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:39.997411 1159 tablet_bootstrap.cc:492] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:39.997592 1159 ts_tablet_manager.cc:1397] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.175s user 0.027s sys 0.054s
I20250629 01:51:39.998291 1159 raft_consensus.cc:357] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.998510 1159 raft_consensus.cc:383] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:39.998615 1159 raft_consensus.cc:738] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:39.998602 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88031f8d8cbc44e4b8c53d9cd5360552" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:39.998775 1052 raft_consensus.cc:2466] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:39.998883 1159 consensus_queue.cc:260] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:39.999341 1159 ts_tablet_manager.cc:1428] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:39.999667 1159 tablet_bootstrap.cc:492] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.000643 850 leader_election.cc:304] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.000991 1173 raft_consensus.cc:2802] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.001124 1173 raft_consensus.cc:491] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.001204 1173 raft_consensus.cc:3058] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.001391 1159 tablet_bootstrap.cc:654] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.003330 1155 tablet_bootstrap.cc:492] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.003728 1173 raft_consensus.cc:513] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.004200 1155 ts_tablet_manager.cc:1397] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.190s user 0.043s sys 0.105s
I20250629 01:51:40.004294 1173 leader_election.cc:290] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.004684 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88031f8d8cbc44e4b8c53d9cd5360552" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:40.004844 1052 raft_consensus.cc:3058] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.005820 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88031f8d8cbc44e4b8c53d9cd5360552" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:40.005945 1155 raft_consensus.cc:357] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.006156 1155 raft_consensus.cc:383] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.006206 777 raft_consensus.cc:2466] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:40.006282 1155 raft_consensus.cc:738] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.006681 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88031f8d8cbc44e4b8c53d9cd5360552" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:40.006803 777 raft_consensus.cc:3058] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.007098 1052 raft_consensus.cc:2466] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:40.007965 1155 consensus_queue.cc:260] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.008256 850 leader_election.cc:304] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.008577 1173 raft_consensus.cc:2802] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.009665 777 raft_consensus.cc:2466] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:40.011337 1155 ts_tablet_manager.cc:1428] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:40.012972 1155 tablet_bootstrap.cc:492] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.014818 1155 tablet_bootstrap.cc:654] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:40.020161 986 leader_election.cc:343] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: b966a9c2749640808cf97059845d3b44
I20250629 01:51:40.020298 986 leader_election.cc:304] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:40.020495 1154 tablet_bootstrap.cc:492] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:51:40.020571 986 raft_consensus.cc:2684] T b966a9c2749640808cf97059845d3b44 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:40.021711 1159 tablet_bootstrap.cc:492] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.021818 1159 ts_tablet_manager.cc:1397] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20250629 01:51:40.022435 1159 raft_consensus.cc:357] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.022571 1159 raft_consensus.cc:383] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.022642 1159 raft_consensus.cc:738] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.022814 1159 consensus_queue.cc:260] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.023209 1159 ts_tablet_manager.cc:1428] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:40.023376 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 64477468c6da4b4ba282d8530eb5cbb5
I20250629 01:51:40.023424 1159 tablet_bootstrap.cc:492] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.023792 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3a45e95a4a6a45f9b7c7b09bd69112b4
I20250629 01:51:40.024751 1159 tablet_bootstrap.cc:654] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.025446 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8603f3907bdd482ea98c45f0c5cd7903" } modify_external_catalogs: true
I20250629 01:51:40.025584 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8603f3907bdd482ea98c45f0c5cd7903" } modify_external_catalogs: true
W20250629 01:51:40.025951 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf consensus state has no leader
W20250629 01:51:40.026273 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc6f34d6b31849e190ebfaf1a273d29c consensus state has no leader
I20250629 01:51:40.026540 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.026716 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 78a5f38231a34efaab177a23cf693d51
I20250629 01:51:40.026726 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.026754 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59bcc265cd41a7852b98dc84b34aff with delete_type TABLET_DATA_DELETED (replaced by tablet 95c14dacdd0d4156ba7b172edf255ae3) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.026914 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.026968 1158 tablet_replica.cc:331] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.027094 754 tablet_service.cc:1515] Processing DeleteTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.027104 1158 raft_consensus.cc:2241] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.027271 1158 raft_consensus.cc:2270] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.027614 1156 tablet_replica.cc:331] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.027786 1156 raft_consensus.cc:2241] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.027969 1156 raft_consensus.cc:2270] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.029335 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.029572 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.029764 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a5f38231a34efaab177a23cf693d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:40.030601 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 64477468c6da4b4ba282d8530eb5cbb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64477468c6da4b4ba282d8530eb5cbb5 already in progress: creating tablet
W20250629 01:51:40.030838 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 already in progress: creating tablet
I20250629 01:51:40.030890 1155 tablet_bootstrap.cc:492] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
W20250629 01:51:40.031029 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 64477468c6da4b4ba282d8530eb5cbb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64477468c6da4b4ba282d8530eb5cbb5 already in progress: creating tablet
I20250629 01:51:40.031039 1155 ts_tablet_manager.cc:1397] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250629 01:51:40.031481 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a5f38231a34efaab177a23cf693d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.031598 1158 ts_tablet_manager.cc:1905] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.031693 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.032053 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a5f38231a34efaab177a23cf693d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.032526 1155 raft_consensus.cc:357] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.032721 1155 raft_consensus.cc:383] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:51:40.033062 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 64477468c6da4b4ba282d8530eb5cbb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64477468c6da4b4ba282d8530eb5cbb5 already in progress: creating tablet
W20250629 01:51:40.033300 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 already in progress: creating tablet
W20250629 01:51:40.033509 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 78a5f38231a34efaab177a23cf693d51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a5f38231a34efaab177a23cf693d51 already in progress: creating tablet
I20250629 01:51:40.034724 1173 raft_consensus.cc:695] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:40.035213 1173 consensus_queue.cc:237] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.035319 1155 raft_consensus.cc:738] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.035632 1155 consensus_queue.cc:260] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.037528 1156 ts_tablet_manager.cc:1905] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.039569 1155 ts_tablet_manager.cc:1428] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250629 01:51:40.041477 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.042954 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b710052005c4a2aaade1dc60409aef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.043922 1155 tablet_bootstrap.cc:492] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.044857 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b3de9ab31fc34ba88ac7616dff6e7d5e
I20250629 01:51:40.045562 888 tablet_service.cc:1515] Processing DeleteTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.045841 754 tablet_service.cc:1515] Processing DeleteTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.046026 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:40.046099 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b3de9ab31fc34ba88ac7616dff6e7d5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3de9ab31fc34ba88ac7616dff6e7d5e already in progress: creating tablet
I20250629 01:51:40.047807 1155 tablet_bootstrap.cc:654] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.053822 754 tablet_service.cc:1468] Processing CreateTablet for tablet cade2a7984a3413a9690af48b8c28e84 (DEFAULT_TABLE table=default.Table_fbbdafc99d2e477b92c7724d35c4f3ac [id=111d4dc551ed44a1a99a4dfcb0916eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.053929 1158 ts_tablet_manager.cc:1918] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.054013 1158 log.cc:1199] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ab59bcc265cd41a7852b98dc84b34aff
I20250629 01:51:40.054172 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cade2a7984a3413a9690af48b8c28e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.054349 1158 ts_tablet_manager.cc:1939] T ab59bcc265cd41a7852b98dc84b34aff P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.055120 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ab59bcc265cd41a7852b98dc84b34aff (table default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]) successfully deleted
I20250629 01:51:40.055326 1154 ts_tablet_manager.cc:1397] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.004s
I20250629 01:51:40.056067 1154 raft_consensus.cc:357] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.056264 1154 raft_consensus.cc:383] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.056327 1154 raft_consensus.cc:738] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.056526 1154 consensus_queue.cc:260] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:40.057443 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 78a5f38231a34efaab177a23cf693d51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a5f38231a34efaab177a23cf693d51 already in progress: creating tablet
W20250629 01:51:40.057898 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 already in progress: creating tablet
I20250629 01:51:40.058056 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_40038832d9f84a35b7db42595981b08d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.058162 1158 tablet_replica.cc:331] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.058312 1158 raft_consensus.cc:2241] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.058427 1158 raft_consensus.cc:2270] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:40.058511 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40038832d9f84a35b7db42595981b08d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.049253 1156 ts_tablet_manager.cc:1918] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.058631 1156 log.cc:1199] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d6fd2102b5b84ae4ae6d58f2a577ee69
I20250629 01:51:40.058954 1156 ts_tablet_manager.cc:1939] T d6fd2102b5b84ae4ae6d58f2a577ee69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.059165 1158 ts_tablet_manager.cc:1905] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.059774 1156 tablet_replica.cc:331] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.060000 1154 ts_tablet_manager.cc:1428] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:40.060344 1154 tablet_bootstrap.cc:492] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.061374 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d6fd2102b5b84ae4ae6d58f2a577ee69 (table default.Table_20f0297464ef4a569b4432d1fc39b500 [id=dc5b44952f4a43c88249de26b060f934]) successfully deleted
I20250629 01:51:40.061628 1156 raft_consensus.cc:2241] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:40.061635 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 78a5f38231a34efaab177a23cf693d51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a5f38231a34efaab177a23cf693d51 already in progress: creating tablet
I20250629 01:51:40.062923 1156 raft_consensus.cc:2270] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.063593 1156 ts_tablet_manager.cc:1905] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.064631 1154 tablet_bootstrap.cc:654] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.068558 888 tablet_service.cc:1468] Processing CreateTablet for tablet cade2a7984a3413a9690af48b8c28e84 (DEFAULT_TABLE table=default.Table_fbbdafc99d2e477b92c7724d35c4f3ac [id=111d4dc551ed44a1a99a4dfcb0916eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.068894 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cade2a7984a3413a9690af48b8c28e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.069236 755 tablet_service.cc:1468] Processing CreateTablet for tablet 02d072bf72044dc48d8090660ecb7d47 (DEFAULT_TABLE table=default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.069526 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d072bf72044dc48d8090660ecb7d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.072305 752 tablet_service.cc:1468] Processing CreateTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.072328 886 tablet_service.cc:1468] Processing CreateTablet for tablet 02d072bf72044dc48d8090660ecb7d47 (DEFAULT_TABLE table=default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.072613 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d072bf72044dc48d8090660ecb7d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.072608 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5b835ff8624ebf8956eeb965ce5950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.076352 894 tablet_service.cc:1468] Processing CreateTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.076654 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5b835ff8624ebf8956eeb965ce5950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.078820 1158 ts_tablet_manager.cc:1918] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.078949 1158 log.cc:1199] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0b710052005c4a2aaade1dc60409aef3
I20250629 01:51:40.079264 1158 ts_tablet_manager.cc:1939] T 0b710052005c4a2aaade1dc60409aef3 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.081545 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 02d072bf72044dc48d8090660ecb7d47 (DEFAULT_TABLE table=default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.081843 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d072bf72044dc48d8090660ecb7d47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.084344 1030 tablet_service.cc:1468] Processing CreateTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.084635 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5b835ff8624ebf8956eeb965ce5950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.088233 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ab05d7564a8476c9f56c06687d11f36 with delete_type TABLET_DATA_DELETED (replaced by tablet 4a22af67c80541b38f2d4c121fdeb6e4) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.088459 1158 tablet_replica.cc:331] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.088639 1158 raft_consensus.cc:2241] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.088814 1158 raft_consensus.cc:2270] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.089982 1032 tablet_service.cc:1468] Processing CreateTablet for tablet cade2a7984a3413a9690af48b8c28e84 (DEFAULT_TABLE table=default.Table_fbbdafc99d2e477b92c7724d35c4f3ac [id=111d4dc551ed44a1a99a4dfcb0916eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.090299 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cade2a7984a3413a9690af48b8c28e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.092872 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0b710052005c4a2aaade1dc60409aef3 (table default.Table_c440da96423948c4bc188695d63152cc [id=64a5b540bb01488c92bb18807a03e2b7]) successfully deleted
W20250629 01:51:40.093106 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b3de9ab31fc34ba88ac7616dff6e7d5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3de9ab31fc34ba88ac7616dff6e7d5e already in progress: creating tablet
I20250629 01:51:40.095086 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_84a213599d1e4030bd7ec42e4bb8c128" } new_table_name: "default.Table_3185dd711b4b48278eb81b5946a06f0f" modify_external_catalogs: true
W20250629 01:51:40.100122 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf consensus state has no leader
I20250629 01:51:40.101475 896 tablet_service.cc:1468] Processing CreateTablet for tablet fb359994b20748518a676045ac6f0df8 (DEFAULT_TABLE table=default.Table_4ff2476a31de4c299bf428196cd197ba [id=4428e6c03ff541c89ad394b08004be0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.101794 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb359994b20748518a676045ac6f0df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.104338 889 tablet_service.cc:1468] Processing CreateTablet for tablet 82d58c88a6994d828b444c5814b287ff (DEFAULT_TABLE table=default.Table_6e7aa21adf4540bcb0f4c7f4e711f90b [id=74626a8a46524de4b96a5e946526aefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.104638 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d58c88a6994d828b444c5814b287ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.108366 893 tablet_service.cc:1468] Processing CreateTablet for tablet caf5826b7e494afc9ae884fe7e94353a (DEFAULT_TABLE table=default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.108676 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf5826b7e494afc9ae884fe7e94353a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.108654 455 catalog_manager.cc:5582] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:40.109407 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:40.112352 892 tablet_service.cc:1468] Processing CreateTablet for tablet fca257d6cb614110948e1045dee44bdc (DEFAULT_TABLE table=default.Table_96f437ef98bb46f69002f7508991afbe [id=ad7989043c924e598bdc640d4deecec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.112650 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca257d6cb614110948e1045dee44bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.115823 891 tablet_service.cc:1468] Processing CreateTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 (DEFAULT_TABLE table=default.Table_76c73b5d51504e688e3d59a43f6ab0a0 [id=05a633be0bb74a1388daba48dd2bfb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.116148 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3bf5fd9b674eeb8a5a029152cbae13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.117713 885 tablet_service.cc:1468] Processing CreateTablet for tablet 582e063f97ab415cab126845789e6e20 (DEFAULT_TABLE table=default.Table_30a1da646c914cde85078e3e1270a01e [id=26e8b18cac084e4c9e603950a6595c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.118281 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 582e063f97ab415cab126845789e6e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.119591 1155 tablet_bootstrap.cc:492] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.119709 1155 ts_tablet_manager.cc:1397] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.039s
I20250629 01:51:40.120420 1155 raft_consensus.cc:357] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.120592 1155 raft_consensus.cc:383] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.120669 1155 raft_consensus.cc:738] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.120592 887 tablet_service.cc:1468] Processing CreateTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe (DEFAULT_TABLE table=default.Table_3ac2dc2474da47f7ba25608e622e9a68 [id=5b172406f58f4023b2b76511690541f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.120862 1155 consensus_queue.cc:260] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.121045 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200871d1dde144eeb59cc271ce9e4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.123459 1155 ts_tablet_manager.cc:1428] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:40.124323 895 tablet_service.cc:1468] Processing CreateTablet for tablet b939a8568ac14c339258af925fcb809b (DEFAULT_TABLE table=default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.124645 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b939a8568ac14c339258af925fcb809b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.125638 1155 tablet_bootstrap.cc:492] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.126598 890 tablet_service.cc:1468] Processing CreateTablet for tablet d3a91740f16744439f55077c4fbadee1 (DEFAULT_TABLE table=default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 [id=cf0d80c2c9e2464db9ded94438df584a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.126904 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a91740f16744439f55077c4fbadee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.127228 1155 tablet_bootstrap.cc:654] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.131260 754 tablet_service.cc:1468] Processing CreateTablet for tablet fb359994b20748518a676045ac6f0df8 (DEFAULT_TABLE table=default.Table_4ff2476a31de4c299bf428196cd197ba [id=4428e6c03ff541c89ad394b08004be0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.131589 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb359994b20748518a676045ac6f0df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.132191 748 tablet_service.cc:1515] Processing DeleteTablet for tablet ee70cbe4d80e41b99234f1e1b001434b with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.132252 755 tablet_service.cc:1468] Processing CreateTablet for tablet caf5826b7e494afc9ae884fe7e94353a (DEFAULT_TABLE table=default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.132498 756 tablet_service.cc:1468] Processing CreateTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe (DEFAULT_TABLE table=default.Table_3ac2dc2474da47f7ba25608e622e9a68 [id=5b172406f58f4023b2b76511690541f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.132280 752 tablet_service.cc:1468] Processing CreateTablet for tablet 82d58c88a6994d828b444c5814b287ff (DEFAULT_TABLE table=default.Table_6e7aa21adf4540bcb0f4c7f4e711f90b [id=74626a8a46524de4b96a5e946526aefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.132791 750 tablet_service.cc:1468] Processing CreateTablet for tablet fca257d6cb614110948e1045dee44bdc (DEFAULT_TABLE table=default.Table_96f437ef98bb46f69002f7508991afbe [id=ad7989043c924e598bdc640d4deecec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.133080 753 tablet_service.cc:1468] Processing CreateTablet for tablet b939a8568ac14c339258af925fcb809b (DEFAULT_TABLE table=default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.132568 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf5826b7e494afc9ae884fe7e94353a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.133415 757 tablet_service.cc:1468] Processing CreateTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 (DEFAULT_TABLE table=default.Table_76c73b5d51504e688e3d59a43f6ab0a0 [id=05a633be0bb74a1388daba48dd2bfb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.133761 751 tablet_service.cc:1468] Processing CreateTablet for tablet d3a91740f16744439f55077c4fbadee1 (DEFAULT_TABLE table=default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 [id=cf0d80c2c9e2464db9ded94438df584a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.134106 749 tablet_service.cc:1468] Processing CreateTablet for tablet 582e063f97ab415cab126845789e6e20 (DEFAULT_TABLE table=default.Table_30a1da646c914cde85078e3e1270a01e [id=26e8b18cac084e4c9e603950a6595c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.135491 1155 tablet_bootstrap.cc:492] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.135649 1155 ts_tablet_manager.cc:1397] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250629 01:51:40.136209 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d58c88a6994d828b444c5814b287ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.136816 1155 raft_consensus.cc:357] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.138478 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b939a8568ac14c339258af925fcb809b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.137030 1155 raft_consensus.cc:383] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.138841 1155 raft_consensus.cc:738] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.139204 1155 consensus_queue.cc:260] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.140370 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200871d1dde144eeb59cc271ce9e4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.141924 1155 ts_tablet_manager.cc:1428] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250629 01:51:40.143249 1155 tablet_bootstrap.cc:492] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.144255 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3bf5fd9b674eeb8a5a029152cbae13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.145074 1155 tablet_bootstrap.cc:654] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.148435 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a91740f16744439f55077c4fbadee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.154755 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 582e063f97ab415cab126845789e6e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.159080 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca257d6cb614110948e1045dee44bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.159624 1155 tablet_bootstrap.cc:492] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.159742 1155 ts_tablet_manager.cc:1397] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20250629 01:51:40.160498 1155 raft_consensus.cc:357] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.160705 1155 raft_consensus.cc:383] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.160791 1155 raft_consensus.cc:738] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.160992 1155 consensus_queue.cc:260] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.161401 1155 ts_tablet_manager.cc:1428] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:40.162369 1155 tablet_bootstrap.cc:492] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.164305 1155 tablet_bootstrap.cc:654] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.166381 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e2a173c972234867a7870c52678fc4bf" } new_table_name: "default.Table_ce8f67fb244e463694b27a5c386da0ca" modify_external_catalogs: true
I20250629 01:51:40.167860 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f1372c29f2b4177af51a749a873b87c" } new_table_name: "default.Table_d31db449adde4809af59d2458b8e6118" modify_external_catalogs: true
I20250629 01:51:40.169072 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 267d9a050fa2414d87a04ba3c91dea03 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.169669 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6700b2fe06164cdea55b01221a20b80e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.170161 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6700b2fe06164cdea55b01221a20b80e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.171319 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e63b2128684f4cfc898dee216184ddfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.171808 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e63b2128684f4cfc898dee216184ddfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.172312 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7b25659fe37948109f17e9e6b15518cc" } modify_external_catalogs: true
I20250629 01:51:40.172474 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7b25659fe37948109f17e9e6b15518cc" } modify_external_catalogs: true
I20250629 01:51:40.173830 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ebdea537fafa4ed88bb3006569ce22c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.174326 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebdea537fafa4ed88bb3006569ce22c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.175451 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_bd59d9e2715745828993ae11d0dd56d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.175904 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd59d9e2715745828993ae11d0dd56d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.177588 1158 ts_tablet_manager.cc:1905] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.181437 1156 ts_tablet_manager.cc:1918] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.181558 1156 log.cc:1199] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/88fdf1f8bb96449ba472990092cfe8cf
I20250629 01:51:40.181869 1156 ts_tablet_manager.cc:1939] T 88fdf1f8bb96449ba472990092cfe8cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.183207 1158 ts_tablet_manager.cc:1918] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.183329 1158 log.cc:1199] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/7ab05d7564a8476c9f56c06687d11f36
I20250629 01:51:40.183706 1158 ts_tablet_manager.cc:1939] T 7ab05d7564a8476c9f56c06687d11f36 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.184705 1154 tablet_bootstrap.cc:492] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.184839 1154 ts_tablet_manager.cc:1397] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20250629 01:51:40.185138 1159 tablet_bootstrap.cc:492] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.185271 1159 ts_tablet_manager.cc:1397] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.162s user 0.061s sys 0.079s
I20250629 01:51:40.185472 1030 tablet_service.cc:1468] Processing CreateTablet for tablet fb359994b20748518a676045ac6f0df8 (DEFAULT_TABLE table=default.Table_4ff2476a31de4c299bf428196cd197ba [id=4428e6c03ff541c89ad394b08004be0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.185781 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb359994b20748518a676045ac6f0df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.185912 1159 raft_consensus.cc:357] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.186060 1159 raft_consensus.cc:383] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.186122 1159 raft_consensus.cc:738] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.186344 1159 consensus_queue.cc:260] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.186406 1031 tablet_service.cc:1468] Processing CreateTablet for tablet caf5826b7e494afc9ae884fe7e94353a (DEFAULT_TABLE table=default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.186744 1159 ts_tablet_manager.cc:1428] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:40.186852 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caf5826b7e494afc9ae884fe7e94353a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.186998 1159 tablet_bootstrap.cc:492] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.188416 1159 tablet_bootstrap.cc:654] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.188362 1022 tablet_service.cc:1468] Processing CreateTablet for tablet fca257d6cb614110948e1045dee44bdc (DEFAULT_TABLE table=default.Table_96f437ef98bb46f69002f7508991afbe [id=ad7989043c924e598bdc640d4deecec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.188653 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca257d6cb614110948e1045dee44bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.189464 1156 tablet_replica.cc:331] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.190204 1156 raft_consensus.cc:2241] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.190367 1156 raft_consensus.cc:2270] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.190938 1156 ts_tablet_manager.cc:1905] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.192021 447 master_service.cc:948] ReplaceTablet: received request to replace tablet cc260579c0da4ed38d9b0168e469c1b5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.192802 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe (DEFAULT_TABLE table=default.Table_3ac2dc2474da47f7ba25608e622e9a68 [id=5b172406f58f4023b2b76511690541f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.193104 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200871d1dde144eeb59cc271ce9e4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.195155 1155 tablet_bootstrap.cc:492] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.195279 1155 ts_tablet_manager.cc:1397] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.013s
I20250629 01:51:40.195856 1155 raft_consensus.cc:357] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:40.196007 1155 raft_consensus.cc:383] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.196062 1155 raft_consensus.cc:738] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.196259 1155 consensus_queue.cc:260] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:40.196676 1155 ts_tablet_manager.cc:1428] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:40.196915 1155 tablet_bootstrap.cc:492] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.199182 1023 tablet_service.cc:1468] Processing CreateTablet for tablet b939a8568ac14c339258af925fcb809b (DEFAULT_TABLE table=default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.199481 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b939a8568ac14c339258af925fcb809b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.199738 1156 ts_tablet_manager.cc:1918] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.199843 1156 log.cc:1199] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ee70cbe4d80e41b99234f1e1b001434b
I20250629 01:51:40.200141 1156 ts_tablet_manager.cc:1939] T ee70cbe4d80e41b99234f1e1b001434b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.201766 1155 tablet_bootstrap.cc:654] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.203949 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 (DEFAULT_TABLE table=default.Table_76c73b5d51504e688e3d59a43f6ab0a0 [id=05a633be0bb74a1388daba48dd2bfb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.204814 894 tablet_service.cc:1515] Processing DeleteTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.205057 1157 tablet_replica.cc:331] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.205240 1157 raft_consensus.cc:2241] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.205407 1157 raft_consensus.cc:2270] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.206085 1157 ts_tablet_manager.cc:1905] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.206854 455 master_service.cc:948] ReplaceTablet: received request to replace tablet c5a1e71ab9a84caea5c917d422336bc8 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.208338 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 7ab05d7564a8476c9f56c06687d11f36 (table default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
I20250629 01:51:40.209141 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 88fdf1f8bb96449ba472990092cfe8cf (table default.Table_4dccf19638764ac494fc06ce3a7ae4bf [id=2a49e00329f14706ae427516ea2072c8]) successfully deleted
W20250629 01:51:40.209331 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b3de9ab31fc34ba88ac7616dff6e7d5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3de9ab31fc34ba88ac7616dff6e7d5e already in progress: creating tablet
W20250629 01:51:40.209443 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6688e9665f24e73b80802d47a80a2cd consensus state has no leader
I20250629 01:51:40.209532 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ee70cbe4d80e41b99234f1e1b001434b (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:40.209658 452 master_service.cc:948] ReplaceTablet: received request to replace tablet d237b9f962024ecb967f86a5730a5039 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.210140 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e6b1ecf098ab4e498391e79cfb1c60b0
W20250629 01:51:40.212368 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5a1e71ab9a84caea5c917d422336bc8 consensus state has no leader
I20250629 01:51:40.212842 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 7ae96f2da36744198e3d38cb07bb47a0 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.214253 894 tablet_service.cc:1515] Processing DeleteTablet for tablet ee70cbe4d80e41b99234f1e1b001434b with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.214534 888 tablet_service.cc:1515] Processing DeleteTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.214792 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3bf5fd9b674eeb8a5a029152cbae13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.218510 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 582e063f97ab415cab126845789e6e20 (DEFAULT_TABLE table=default.Table_30a1da646c914cde85078e3e1270a01e [id=26e8b18cac084e4c9e603950a6595c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.218850 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 582e063f97ab415cab126845789e6e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.220371 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d3a91740f16744439f55077c4fbadee1 (DEFAULT_TABLE table=default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 [id=cf0d80c2c9e2464db9ded94438df584a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.220698 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a91740f16744439f55077c4fbadee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.230739 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_358f4795c7e244f2b72efe50dacd310f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.231222 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_358f4795c7e244f2b72efe50dacd310f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.231722 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2ce50be1c3444c059ed91faaf9cd5854"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.232168 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ce50be1c3444c059ed91faaf9cd5854 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.233208 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_534d46b1a5ee4435844d4dd7c6626d1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.233654 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_534d46b1a5ee4435844d4dd7c6626d1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.234633 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8917f34339fb40e0a110ff85060ad961" } new_table_name: "default.Table_0019f37bf9d34f969b4ef89adc5dfb9d" modify_external_catalogs: true
I20250629 01:51:40.235747 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_654038c5004a437482989db6feea6573" } modify_external_catalogs: true
I20250629 01:51:40.235893 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_654038c5004a437482989db6feea6573" } modify_external_catalogs: true
I20250629 01:51:40.236285 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_38d08e8c1c0d4beea36513f66e9cbe43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.236716 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38d08e8c1c0d4beea36513f66e9cbe43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.237718 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ce8f67fb244e463694b27a5c386da0ca" } new_table_name: "default.Table_aa33973948ab4d68b062deca5d60a668" modify_external_catalogs: true
I20250629 01:51:40.238824 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e4165d1f8e364693a7dd96d2e5e47fa2" } modify_external_catalogs: true
I20250629 01:51:40.238978 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e4165d1f8e364693a7dd96d2e5e47fa2" } modify_external_catalogs: true
I20250629 01:51:40.244194 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c5a1e71ab9a84caea5c917d422336bc8
I20250629 01:51:40.244602 455 catalog_manager.cc:6652] ReplaceTablet: tablet c5a1e71ab9a84caea5c917d422336bc8 deleted and replaced by tablet 97439c6a6e844610b55b899966ae5b67
I20250629 01:51:40.244895 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3ac2dc2474da47f7ba25608e622e9a68" } modify_external_catalogs: true
I20250629 01:51:40.245052 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3ac2dc2474da47f7ba25608e622e9a68" } modify_external_catalogs: true
I20250629 01:51:40.250028 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9edf5b31c7b24325930accee8cb627c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.250484 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9edf5b31c7b24325930accee8cb627c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.253568 1154 raft_consensus.cc:357] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.254192 1154 raft_consensus.cc:383] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.254428 1154 raft_consensus.cc:738] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.254855 1154 consensus_queue.cc:260] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.255421 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4b11b5c3ee064217b3fcbeb9ad01293c" } modify_external_catalogs: true
I20250629 01:51:40.255563 1157 ts_tablet_manager.cc:1918] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.255560 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4b11b5c3ee064217b3fcbeb9ad01293c" } modify_external_catalogs: true
I20250629 01:51:40.255677 1157 log.cc:1199] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d6fd2102b5b84ae4ae6d58f2a577ee69
I20250629 01:51:40.256047 1157 ts_tablet_manager.cc:1939] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.257941 750 tablet_service.cc:1515] Processing DeleteTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 97439c6a6e844610b55b899966ae5b67) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.258611 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 82d58c88a6994d828b444c5814b287ff (DEFAULT_TABLE table=default.Table_6e7aa21adf4540bcb0f4c7f4e711f90b [id=74626a8a46524de4b96a5e946526aefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.258930 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d58c88a6994d828b444c5814b287ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.260509 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 05905638c6e241a095438f277f53adab (DEFAULT_TABLE table=default.Table_40038832d9f84a35b7db42595981b08d [id=151987e142c54e60b5137314e329dd82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.260917 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05905638c6e241a095438f277f53adab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.263075 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 97439c6a6e844610b55b899966ae5b67) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.263724 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.264545 1154 ts_tablet_manager.cc:1428] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.080s user 0.002s sys 0.000s
I20250629 01:51:40.268636 751 tablet_service.cc:1468] Processing CreateTablet for tablet 05905638c6e241a095438f277f53adab (DEFAULT_TABLE table=default.Table_40038832d9f84a35b7db42595981b08d [id=151987e142c54e60b5137314e329dd82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.269052 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05905638c6e241a095438f277f53adab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.276538 757 tablet_service.cc:1515] Processing DeleteTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:40.277554 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c5a1e71ab9a84caea5c917d422336bc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5a1e71ab9a84caea5c917d422336bc8 already in progress: creating tablet
W20250629 01:51:40.278004 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e6b1ecf098ab4e498391e79cfb1c60b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6b1ecf098ab4e498391e79cfb1c60b0 already in progress: creating tablet
W20250629 01:51:40.278355 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c5a1e71ab9a84caea5c917d422336bc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5a1e71ab9a84caea5c917d422336bc8 already in progress: creating tablet
W20250629 01:51:40.278574 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e6b1ecf098ab4e498391e79cfb1c60b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6b1ecf098ab4e498391e79cfb1c60b0 already in progress: creating tablet
I20250629 01:51:40.278787 1154 tablet_bootstrap.cc:492] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.279486 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet d6fd2102b5b84ae4ae6d58f2a577ee69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.281692 1157 tablet_replica.cc:331] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.281824 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d6fd2102b5b84ae4ae6d58f2a577ee69 (table default.Table_20f0297464ef4a569b4432d1fc39b500 [id=dc5b44952f4a43c88249de26b060f934]) successfully deleted
W20250629 01:51:40.282374 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57f7f71ab5e54d249141c89d9e806111 consensus state has no leader
I20250629 01:51:40.282490 1158 tablet_replica.cc:331] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.282644 1158 raft_consensus.cc:2241] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.282778 1158 raft_consensus.cc:2270] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.268357 892 tablet_service.cc:1515] Processing DeleteTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 97439c6a6e844610b55b899966ae5b67) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.283275 1157 raft_consensus.cc:2241] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.286435 1164 raft_consensus.cc:491] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:40.286589 1164 raft_consensus.cc:513] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.286906 1157 raft_consensus.cc:2270] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.287153 1164 leader_election.cc:290] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.287448 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1185e85345664dfd92c134c5552e7d48" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:40.287585 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1185e85345664dfd92c134c5552e7d48" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:40.287917 712 leader_election.cc:343] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:40.288127 710 leader_election.cc:343] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:40.288287 710 leader_election.cc:304] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:40.277184 891 tablet_service.cc:1468] Processing CreateTablet for tablet 05905638c6e241a095438f277f53adab (DEFAULT_TABLE table=default.Table_40038832d9f84a35b7db42595981b08d [id=151987e142c54e60b5137314e329dd82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.288677 1164 raft_consensus.cc:2747] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:40.268232 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.289014 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05905638c6e241a095438f277f53adab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.289732 1157 ts_tablet_manager.cc:1905] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.280441 1154 tablet_bootstrap.cc:654] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.291185 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_47bdd31ed10f4b8183ea24a4fc30d77e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.291625 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47bdd31ed10f4b8183ea24a4fc30d77e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.294668 1154 tablet_bootstrap.cc:492] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.294826 1154 ts_tablet_manager.cc:1397] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250629 01:51:40.295255 1158 ts_tablet_manager.cc:1905] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.300733 1154 raft_consensus.cc:357] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.300956 1154 raft_consensus.cc:383] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.301016 1154 raft_consensus.cc:738] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.301659 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fbbdafc99d2e477b92c7724d35c4f3ac" } new_table_name: "default.Table_cdb1b42c2c774292b4ee2e323ddee5f6" modify_external_catalogs: true
I20250629 01:51:40.303043 1154 consensus_queue.cc:260] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:40.303787 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97439c6a6e844610b55b899966ae5b67 has no consensus state
I20250629 01:51:40.304126 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d3a91740f16744439f55077c4fbadee1
I20250629 01:51:40.306654 777 raft_consensus.cc:1273] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:40.307103 1052 raft_consensus.cc:1273] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:40.307341 1173 consensus_queue.cc:1035] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:40.308147 1155 tablet_bootstrap.cc:492] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.308279 1155 ts_tablet_manager.cc:1397] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.111s user 0.020s sys 0.032s
I20250629 01:51:40.309022 1155 raft_consensus.cc:357] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.309198 1155 raft_consensus.cc:383] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.309300 1155 raft_consensus.cc:738] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.309496 1155 consensus_queue.cc:260] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.309705 1173 consensus_queue.cc:1035] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:40.309760 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a91740f16744439f55077c4fbadee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:40.317523 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cade2a7984a3413a9690af48b8c28e84 consensus state has no leader
I20250629 01:51:40.319272 1154 ts_tablet_manager.cc:1428] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20250629 01:51:40.319568 1159 tablet_bootstrap.cc:492] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.319694 1159 ts_tablet_manager.cc:1397] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.133s user 0.021s sys 0.038s
I20250629 01:51:40.320474 1154 tablet_bootstrap.cc:492] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.320397 1159 raft_consensus.cc:357] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.320554 1159 raft_consensus.cc:383] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.320626 1159 raft_consensus.cc:738] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.320814 1159 consensus_queue.cc:260] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.322031 1159 ts_tablet_manager.cc:1428] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:40.322497 1159 tablet_bootstrap.cc:492] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.324599 1158 ts_tablet_manager.cc:1918] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.324662 1159 tablet_bootstrap.cc:654] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.324826 1158 log.cc:1199] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d6fd2102b5b84ae4ae6d58f2a577ee69
I20250629 01:51:40.325150 1158 ts_tablet_manager.cc:1939] T d6fd2102b5b84ae4ae6d58f2a577ee69 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.325990 1154 tablet_bootstrap.cc:654] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.329264 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a91740f16744439f55077c4fbadee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.330098 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:40.330642 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 88fdf1f8bb96449ba472990092cfe8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.330943 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (table default.Table_3185dd711b4b48278eb81b5946a06f0f [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 3 got 0
I20250629 01:51:40.331797 450 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]). Expected version 2 got 0
I20250629 01:51:40.333039 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (table default.Table_3185dd711b4b48278eb81b5946a06f0f [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 3 got 0
W20250629 01:51:40.333882 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d3a91740f16744439f55077c4fbadee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a91740f16744439f55077c4fbadee1 already in progress: creating tablet
I20250629 01:51:40.334116 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d6fd2102b5b84ae4ae6d58f2a577ee69 (table default.Table_20f0297464ef4a569b4432d1fc39b500 [id=dc5b44952f4a43c88249de26b060f934]) successfully deleted
I20250629 01:51:40.336297 1158 tablet_replica.cc:331] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.336745 1157 ts_tablet_manager.cc:1918] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.337224 1157 log.cc:1199] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/ee70cbe4d80e41b99234f1e1b001434b
I20250629 01:51:40.337607 1157 ts_tablet_manager.cc:1939] T ee70cbe4d80e41b99234f1e1b001434b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.343654 886 tablet_service.cc:1468] Processing CreateTablet for tablet 648fd1ca8f514ea3ba4554452b19933f (DEFAULT_TABLE table=default.Table_6700b2fe06164cdea55b01221a20b80e [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.344329 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet ee70cbe4d80e41b99234f1e1b001434b (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:40.344307 892 tablet_service.cc:1468] Processing CreateTablet for tablet f98bb48081984d23af55d7fa6a8bec7b (DEFAULT_TABLE table=default.Table_ebdea537fafa4ed88bb3006569ce22c9 [id=85d2dfba5d5c422a9631994a62cfe443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.344569 1158 raft_consensus.cc:2241] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.344705 1158 raft_consensus.cc:2270] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.345167 889 tablet_service.cc:1468] Processing CreateTablet for tablet 575006c12f7c41658e234f7c6c425d69 (DEFAULT_TABLE table=default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.345412 1158 ts_tablet_manager.cc:1905] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.345788 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf (DEFAULT_TABLE table=default.Table_bd59d9e2715745828993ae11d0dd56d0 [id=9e7864d3513b4aca8f2b2b2cf84805bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.345945 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648fd1ca8f514ea3ba4554452b19933f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.346177 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f6de505a34afbae48a373b4b2a0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.347784 1159 tablet_bootstrap.cc:492] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.347934 1159 ts_tablet_manager.cc:1397] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.006s
I20250629 01:51:40.348596 1159 raft_consensus.cc:357] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.348748 1159 raft_consensus.cc:383] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.348806 1159 raft_consensus.cc:738] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.348999 1159 consensus_queue.cc:260] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.349555 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98bb48081984d23af55d7fa6a8bec7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.349694 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 200871d1dde144eeb59cc271ce9e4cfe
I20250629 01:51:40.351169 1247 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:40.351439 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.352195 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575006c12f7c41658e234f7c6c425d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.352322 751 tablet_service.cc:1468] Processing CreateTablet for tablet 648fd1ca8f514ea3ba4554452b19933f (DEFAULT_TABLE table=default.Table_6700b2fe06164cdea55b01221a20b80e [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.352624 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648fd1ca8f514ea3ba4554452b19933f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.353345 1159 ts_tablet_manager.cc:1428] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:40.353619 1159 tablet_bootstrap.cc:492] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.354991 1159 tablet_bootstrap.cc:654] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.360414 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 648fd1ca8f514ea3ba4554452b19933f (DEFAULT_TABLE table=default.Table_6700b2fe06164cdea55b01221a20b80e [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.362753 1030 tablet_service.cc:1468] Processing CreateTablet for tablet f98bb48081984d23af55d7fa6a8bec7b (DEFAULT_TABLE table=default.Table_ebdea537fafa4ed88bb3006569ce22c9 [id=85d2dfba5d5c422a9631994a62cfe443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.363500 757 tablet_service.cc:1468] Processing CreateTablet for tablet f98bb48081984d23af55d7fa6a8bec7b (DEFAULT_TABLE table=default.Table_ebdea537fafa4ed88bb3006569ce22c9 [id=85d2dfba5d5c422a9631994a62cfe443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.363829 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98bb48081984d23af55d7fa6a8bec7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.368832 751 tablet_service.cc:1468] Processing CreateTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf (DEFAULT_TABLE table=default.Table_bd59d9e2715745828993ae11d0dd56d0 [id=9e7864d3513b4aca8f2b2b2cf84805bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.369271 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f6de505a34afbae48a373b4b2a0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.369496 1157 tablet_replica.cc:331] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.370308 1157 raft_consensus.cc:2241] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.370687 1159 tablet_bootstrap.cc:492] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.370721 1157 raft_consensus.cc:2270] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.371397 1248 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:40.373529 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:40.370821 1159 ts_tablet_manager.cc:1397] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
W20250629 01:51:40.376567 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e6b1ecf098ab4e498391e79cfb1c60b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6b1ecf098ab4e498391e79cfb1c60b0 already in progress: creating tablet
I20250629 01:51:40.376849 1159 raft_consensus.cc:357] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.376991 1159 raft_consensus.cc:383] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.377072 1159 raft_consensus.cc:738] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:51:40.377099 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c5a1e71ab9a84caea5c917d422336bc8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5a1e71ab9a84caea5c917d422336bc8 already in progress: creating tablet
I20250629 01:51:40.377265 1159 consensus_queue.cc:260] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.377878 1159 ts_tablet_manager.cc:1428] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:40.377491 1157 ts_tablet_manager.cc:1905] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.378150 1159 tablet_bootstrap.cc:492] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.376152 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf (DEFAULT_TABLE table=default.Table_bd59d9e2715745828993ae11d0dd56d0 [id=9e7864d3513b4aca8f2b2b2cf84805bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.379782 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 575006c12f7c41658e234f7c6c425d69 (DEFAULT_TABLE table=default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.380153 1159 tablet_bootstrap.cc:654] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.380889 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.381108 1155 ts_tablet_manager.cc:1428] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.073s user 0.010s sys 0.011s
I20250629 01:51:40.381403 1155 tablet_bootstrap.cc:492] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.381762 753 tablet_service.cc:1468] Processing CreateTablet for tablet 575006c12f7c41658e234f7c6c425d69 (DEFAULT_TABLE table=default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.382253 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575006c12f7c41658e234f7c6c425d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.382932 1155 tablet_bootstrap.cc:654] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.385576 1157 ts_tablet_manager.cc:1918] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.385736 1157 log.cc:1199] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/88fdf1f8bb96449ba472990092cfe8cf
I20250629 01:51:40.386157 1157 ts_tablet_manager.cc:1939] T 88fdf1f8bb96449ba472990092cfe8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.388182 1159 tablet_bootstrap.cc:492] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.388311 1159 ts_tablet_manager.cc:1397] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
W20250629 01:51:40.389350 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 200871d1dde144eeb59cc271ce9e4cfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 200871d1dde144eeb59cc271ce9e4cfe already in progress: creating tablet
I20250629 01:51:40.388867 1159 raft_consensus.cc:357] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.389483 1159 raft_consensus.cc:383] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.389556 1159 raft_consensus.cc:738] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.389539 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.389762 1159 consensus_queue.cc:260] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.390370 1159 ts_tablet_manager.cc:1428] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:40.389721 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 88fdf1f8bb96449ba472990092cfe8cf (table default.Table_4dccf19638764ac494fc06ce3a7ae4bf [id=2a49e00329f14706ae427516ea2072c8]) successfully deleted
I20250629 01:51:40.390621 1159 tablet_bootstrap.cc:492] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:40.390726 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d3a91740f16744439f55077c4fbadee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a91740f16744439f55077c4fbadee1 already in progress: creating tablet
W20250629 01:51:40.390990 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 200871d1dde144eeb59cc271ce9e4cfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 200871d1dde144eeb59cc271ce9e4cfe already in progress: creating tablet
I20250629 01:51:40.341562 1154 tablet_bootstrap.cc:492] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.392164 1154 ts_tablet_manager.cc:1397] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20250629 01:51:40.392906 1154 raft_consensus.cc:357] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:40.393106 1154 raft_consensus.cc:383] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.393229 1159 tablet_bootstrap.cc:654] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.393246 1154 raft_consensus.cc:738] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.393527 1154 consensus_queue.cc:260] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:40.394492 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a229b4f9d19147a5990f799d1dc0e5d7
I20250629 01:51:40.395116 1155 tablet_bootstrap.cc:492] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.395334 1155 ts_tablet_manager.cc:1397] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20250629 01:51:40.395596 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 97e03c52aeb4432cab002abdae5109ba
I20250629 01:51:40.396066 1154 ts_tablet_manager.cc:1428] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:40.396477 1155 raft_consensus.cc:357] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.396667 1155 raft_consensus.cc:383] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.396732 1155 raft_consensus.cc:738] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.396818 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]). Expected version 2 got 0
I20250629 01:51:40.396931 1155 consensus_queue.cc:260] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.397454 1155 ts_tablet_manager.cc:1428] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:40.397740 1155 tablet_bootstrap.cc:492] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.397969 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.398777 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.399061 1155 tablet_bootstrap.cc:654] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.399295 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e03c52aeb4432cab002abdae5109ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.400039 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e03c52aeb4432cab002abdae5109ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:40.400337 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a229b4f9d19147a5990f799d1dc0e5d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a229b4f9d19147a5990f799d1dc0e5d7 already in progress: creating tablet
W20250629 01:51:40.400559 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 97e03c52aeb4432cab002abdae5109ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97e03c52aeb4432cab002abdae5109ba already in progress: creating tablet
I20250629 01:51:40.401690 1154 tablet_bootstrap.cc:492] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.402668 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.401428 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]). Expected version 2 got 0
I20250629 01:51:40.402228 451 master_service.cc:948] ReplaceTablet: received request to replace tablet f7bb791d4a38481aa05a4dbbe3577461 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.404814 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e03c52aeb4432cab002abdae5109ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:40.407078 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a229b4f9d19147a5990f799d1dc0e5d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a229b4f9d19147a5990f799d1dc0e5d7 already in progress: creating tablet
W20250629 01:51:40.408210 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 97e03c52aeb4432cab002abdae5109ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97e03c52aeb4432cab002abdae5109ba already in progress: creating tablet
I20250629 01:51:40.409337 449 master_service.cc:948] ReplaceTablet: received request to replace tablet f98bb48081984d23af55d7fa6a8bec7b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.410056 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98bb48081984d23af55d7fa6a8bec7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.410669 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f6de505a34afbae48a373b4b2a0cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.412592 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a91740f16744439f55077c4fbadee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.413594 1155 tablet_bootstrap.cc:492] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.413708 1155 ts_tablet_manager.cc:1397] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.010s
I20250629 01:51:40.414342 1155 raft_consensus.cc:357] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.414637 1155 raft_consensus.cc:383] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.414710 1155 raft_consensus.cc:738] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.414880 1155 consensus_queue.cc:260] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.415333 1155 ts_tablet_manager.cc:1428] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:40.415638 1155 tablet_bootstrap.cc:492] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.417075 1154 tablet_bootstrap.cc:654] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.417378 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 575006c12f7c41658e234f7c6c425d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.418514 1158 ts_tablet_manager.cc:1918] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.418627 1158 log.cc:1199] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/88fdf1f8bb96449ba472990092cfe8cf
I20250629 01:51:40.418987 1158 ts_tablet_manager.cc:1939] T 88fdf1f8bb96449ba472990092cfe8cf P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.420641 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648fd1ca8f514ea3ba4554452b19933f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.421262 1155 tablet_bootstrap.cc:654] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.425837 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6700b2fe06164cdea55b01221a20b80e" } new_table_name: "default.Table_ed3568508cfb4c4b9297f986d7a9963e" modify_external_catalogs: true
I20250629 01:51:40.427203 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_adb2b0b91c94485fa3ede26987189123" } new_table_name: "default.Table_1fb69ddfd0b2426abea3a0352dddb3a7" modify_external_catalogs: true
I20250629 01:51:40.428385 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.428910 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.430025 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_40038832d9f84a35b7db42595981b08d" } modify_external_catalogs: true
I20250629 01:51:40.430213 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_40038832d9f84a35b7db42595981b08d" } modify_external_catalogs: true
I20250629 01:51:40.430732 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6b615ab3e79749aab1024e49691c3762"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.431207 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b615ab3e79749aab1024e49691c3762 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.432341 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b406b018163a43c38fcb2ccfb90c9607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.432793 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b406b018163a43c38fcb2ccfb90c9607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.433717 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_97eb293848a64879af553f06d6660c31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.434150 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97eb293848a64879af553f06d6660c31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.435000 1155 tablet_bootstrap.cc:492] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.435143 1155 ts_tablet_manager.cc:1397] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.008s
I20250629 01:51:40.435783 1155 raft_consensus.cc:357] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.435945 1155 raft_consensus.cc:383] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.436012 1155 raft_consensus.cc:738] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.436250 1155 consensus_queue.cc:260] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.436628 1155 ts_tablet_manager.cc:1428] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:40.437773 1155 tablet_bootstrap.cc:492] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.439421 1154 tablet_bootstrap.cc:492] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.439569 1154 ts_tablet_manager.cc:1397] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.008s
I20250629 01:51:40.440220 1154 raft_consensus.cc:357] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.440394 1154 raft_consensus.cc:383] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.440481 1154 raft_consensus.cc:738] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.442417 1155 tablet_bootstrap.cc:654] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.441118 1154 consensus_queue.cc:260] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.444741 1154 ts_tablet_manager.cc:1428] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:40.446264 1259 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:40.448376 1154 tablet_bootstrap.cc:492] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.449893 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f7bb791d4a38481aa05a4dbbe3577461
I20250629 01:51:40.450001 1155 tablet_bootstrap.cc:492] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.450098 1155 ts_tablet_manager.cc:1397] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250629 01:51:40.450263 1154 tablet_bootstrap.cc:654] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.450340 451 catalog_manager.cc:6652] ReplaceTablet: tablet f7bb791d4a38481aa05a4dbbe3577461 deleted and replaced by tablet 36fcb819c61c4fabbb0da295d0002998
I20250629 01:51:40.450699 1155 raft_consensus.cc:357] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.450840 1155 raft_consensus.cc:383] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.450924 1155 raft_consensus.cc:738] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.451109 1155 consensus_queue.cc:260] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.451687 1159 tablet_bootstrap.cc:492] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.451831 1159 ts_tablet_manager.cc:1397] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.004s
I20250629 01:51:40.452689 1159 raft_consensus.cc:357] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.452883 1159 raft_consensus.cc:383] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.452976 1159 raft_consensus.cc:738] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.453193 1159 consensus_queue.cc:260] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.453718 1159 ts_tablet_manager.cc:1428] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:40.453919 1155 ts_tablet_manager.cc:1428] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:40.454013 1159 tablet_bootstrap.cc:492] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.455621 1159 tablet_bootstrap.cc:654] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.459309 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 with delete_type TABLET_DATA_DELETED (replaced by tablet 36fcb819c61c4fabbb0da295d0002998) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:40.459908 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f7bb791d4a38481aa05a4dbbe3577461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7bb791d4a38481aa05a4dbbe3577461 already in progress: creating tablet
I20250629 01:51:40.461308 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 88fdf1f8bb96449ba472990092cfe8cf (table default.Table_4dccf19638764ac494fc06ce3a7ae4bf [id=2a49e00329f14706ae427516ea2072c8]) successfully deleted
W20250629 01:51:40.461548 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 200871d1dde144eeb59cc271ce9e4cfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 200871d1dde144eeb59cc271ce9e4cfe already in progress: creating tablet
W20250629 01:51:40.461763 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a229b4f9d19147a5990f799d1dc0e5d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a229b4f9d19147a5990f799d1dc0e5d7 already in progress: creating tablet
W20250629 01:51:40.461947 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 97e03c52aeb4432cab002abdae5109ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97e03c52aeb4432cab002abdae5109ba already in progress: creating tablet
W20250629 01:51:40.462124 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d3a91740f16744439f55077c4fbadee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3a91740f16744439f55077c4fbadee1 already in progress: creating tablet
I20250629 01:51:40.463104 751 tablet_service.cc:1515] Processing DeleteTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 with delete_type TABLET_DATA_DELETED (replaced by tablet 36fcb819c61c4fabbb0da295d0002998) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.463346 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 with delete_type TABLET_DATA_DELETED (replaced by tablet 36fcb819c61c4fabbb0da295d0002998) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.463490 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f98bb48081984d23af55d7fa6a8bec7b
I20250629 01:51:40.463820 449 catalog_manager.cc:6652] ReplaceTablet: tablet f98bb48081984d23af55d7fa6a8bec7b deleted and replaced by tablet 5f60c48c924d49a186a8c81d154bdc54
W20250629 01:51:40.464335 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648fd1ca8f514ea3ba4554452b19933f consensus state has no leader
I20250629 01:51:40.464335 1241 raft_consensus.cc:491] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:40.464444 1241 raft_consensus.cc:513] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.464584 452 master_service.cc:948] ReplaceTablet: received request to replace tablet bd5b835ff8624ebf8956eeb965ce5950 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.465013 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 97e03c52aeb4432cab002abdae5109ba from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.465057 1241 leader_election.cc:290] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.465543 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d237b9f962024ecb967f86a5730a5039" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:40.465553 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b5425ba83ec64461aa18b0a9c4b84977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.465736 916 raft_consensus.cc:2466] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:51:40.465966 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5425ba83ec64461aa18b0a9c4b84977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.466090 712 leader_election.cc:304] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:40.466470 1241 raft_consensus.cc:2802] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.466571 1241 raft_consensus.cc:491] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.466641 1241 raft_consensus.cc:3058] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.466698 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d237b9f962024ecb967f86a5730a5039" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:40.466884 1052 raft_consensus.cc:2466] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:40.467204 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6deb0f6b87e7489d994f1b651f6b5acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.467607 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6deb0f6b87e7489d994f1b651f6b5acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.468716 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f2bc100d78a348c88ed48ed66870200b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.468993 1241 raft_consensus.cc:513] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:40.469127 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2bc100d78a348c88ed48ed66870200b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.469774 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d237b9f962024ecb967f86a5730a5039" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:40.469933 916 raft_consensus.cc:3058] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:40.470104 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3c896a37d9749c5a46bfc7cf7c87911 consensus state has no leader
I20250629 01:51:40.472486 916 raft_consensus.cc:2466] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:40.472824 712 leader_election.cc:304] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:40.473045 1164 raft_consensus.cc:2802] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.473268 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.474067 1241 leader_election.cc:290] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.474128 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.474064 751 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.474838 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d237b9f962024ecb967f86a5730a5039" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:40.475003 1052 raft_consensus.cc:3058] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.478076 1052 raft_consensus.cc:2466] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:40.480037 1155 tablet_bootstrap.cc:492] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:40.480027 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f7bb791d4a38481aa05a4dbbe3577461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7bb791d4a38481aa05a4dbbe3577461 already in progress: creating tablet
W20250629 01:51:40.480384 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f98bb48081984d23af55d7fa6a8bec7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98bb48081984d23af55d7fa6a8bec7b already in progress: creating tablet
W20250629 01:51:40.480643 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f7bb791d4a38481aa05a4dbbe3577461 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7bb791d4a38481aa05a4dbbe3577461 already in progress: creating tablet
W20250629 01:51:40.480896 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f98bb48081984d23af55d7fa6a8bec7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98bb48081984d23af55d7fa6a8bec7b already in progress: creating tablet
I20250629 01:51:40.481339 1164 raft_consensus.cc:695] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:40.482091 1155 tablet_bootstrap.cc:654] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.482285 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.482029 1164 consensus_queue.cc:237] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.491242 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a3096bcff0004df0a66caa7d3fbc796f" } new_table_name: "default.Table_d219b9a9062745bca33e3a0fcf553be0" modify_external_catalogs: true
I20250629 01:51:40.492802 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_115462b9624b4cdb8e39a8f38af93416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.493201 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_115462b9624b4cdb8e39a8f38af93416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.495296 1159 tablet_bootstrap.cc:492] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.495440 1159 ts_tablet_manager.cc:1397] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.014s
W20250629 01:51:40.495479 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f98bb48081984d23af55d7fa6a8bec7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98bb48081984d23af55d7fa6a8bec7b already in progress: creating tablet
I20250629 01:51:40.495815 1155 tablet_bootstrap.cc:492] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.495941 1155 ts_tablet_manager.cc:1397] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.010s
I20250629 01:51:40.496109 1159 raft_consensus.cc:357] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.496274 1159 raft_consensus.cc:383] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.496349 1154 tablet_bootstrap.cc:492] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.496347 1159 raft_consensus.cc:738] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.496467 1154 ts_tablet_manager.cc:1397] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.013s
I20250629 01:51:40.496587 1159 consensus_queue.cc:260] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.496574 1155 raft_consensus.cc:357] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.497028 1155 raft_consensus.cc:383] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.497112 1155 raft_consensus.cc:738] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.497094 1154 raft_consensus.cc:357] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.497227 1154 raft_consensus.cc:383] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.497311 1154 raft_consensus.cc:738] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.497329 1155 consensus_queue.cc:260] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.497506 1154 consensus_queue.cc:260] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.498046 1159 ts_tablet_manager.cc:1428] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:40.498414 1159 tablet_bootstrap.cc:492] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.498730 1154 ts_tablet_manager.cc:1428] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:40.499711 1155 ts_tablet_manager.cc:1428] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250629 01:51:40.501329 1155 tablet_bootstrap.cc:492] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.502183 1159 tablet_bootstrap.cc:654] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.502686 1155 tablet_bootstrap.cc:654] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.502737 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]). Expected version 2 got 0
I20250629 01:51:40.503079 1154 tablet_bootstrap.cc:492] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.505132 1154 tablet_bootstrap.cc:654] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.506870 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4ff2476a31de4c299bf428196cd197ba" } new_table_name: "default.Table_a23d7cbbf7804d2e95f76b62ec45c158" modify_external_catalogs: true
I20250629 01:51:40.508281 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5425ba83ec64461aa18b0a9c4b84977" } modify_external_catalogs: true
I20250629 01:51:40.508455 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5425ba83ec64461aa18b0a9c4b84977" } modify_external_catalogs: true
I20250629 01:51:40.509810 1159 tablet_bootstrap.cc:492] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.509955 1159 ts_tablet_manager.cc:1397] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20250629 01:51:40.510661 751 tablet_service.cc:1468] Processing CreateTablet for tablet 7d7b8cf118eb4a5c997815b97b9b7d00 (DEFAULT_TABLE table=default.Table_47bdd31ed10f4b8183ea24a4fc30d77e [id=e6d6685d64154ba8ab80acf601e83986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.510771 1159 raft_consensus.cc:357] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.510944 1159 raft_consensus.cc:383] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.511027 1159 raft_consensus.cc:738] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.511061 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7b8cf118eb4a5c997815b97b9b7d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.511268 1159 consensus_queue.cc:260] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.513024 1159 ts_tablet_manager.cc:1428] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:40.513237 1159 tablet_bootstrap.cc:492] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.516788 1159 tablet_bootstrap.cc:654] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.518364 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 7d7b8cf118eb4a5c997815b97b9b7d00 (DEFAULT_TABLE table=default.Table_47bdd31ed10f4b8183ea24a4fc30d77e [id=e6d6685d64154ba8ab80acf601e83986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.518769 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7b8cf118eb4a5c997815b97b9b7d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.519775 753 tablet_service.cc:1468] Processing CreateTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 (DEFAULT_TABLE table=default.Table_38d08e8c1c0d4beea36513f66e9cbe43 [id=e12703cc30a94230b4c5f0fc21988359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.521023 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3185dd711b4b48278eb81b5946a06f0f" } new_table_name: "default.Table_921f95200675447782407805dfd9fd4a" modify_external_catalogs: true
I20250629 01:51:40.521982 1227 raft_consensus.cc:491] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:40.522096 1227 raft_consensus.cc:513] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.522600 1227 leader_election.cc:290] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.523072 1154 tablet_bootstrap.cc:492] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.523247 1154 ts_tablet_manager.cc:1397] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.013s
I20250629 01:51:40.523756 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 (DEFAULT_TABLE table=default.Table_38d08e8c1c0d4beea36513f66e9cbe43 [id=e12703cc30a94230b4c5f0fc21988359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.524106 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1938e67acaf4ab19fcbed13e1f1bfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.524312 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1185e85345664dfd92c134c5552e7d48" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:40.525692 850 leader_election.cc:343] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:40.526754 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 (DEFAULT_TABLE table=default.Table_358f4795c7e244f2b72efe50dacd310f [id=00b7c47fc4684174b514feaab3b7d411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.527148 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829c64e2c71347f4af2c8870cdd8af31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.535179 1154 raft_consensus.cc:357] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.535388 1154 raft_consensus.cc:383] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.535465 1154 raft_consensus.cc:738] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.535712 1154 consensus_queue.cc:260] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.537133 891 tablet_service.cc:1468] Processing CreateTablet for tablet 7d7b8cf118eb4a5c997815b97b9b7d00 (DEFAULT_TABLE table=default.Table_47bdd31ed10f4b8183ea24a4fc30d77e [id=e6d6685d64154ba8ab80acf601e83986]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.537307 1154 ts_tablet_manager.cc:1428] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250629 01:51:40.537542 1154 tablet_bootstrap.cc:492] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.537561 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d7b8cf118eb4a5c997815b97b9b7d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.541181 889 tablet_service.cc:1468] Processing CreateTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 (DEFAULT_TABLE table=default.Table_38d08e8c1c0d4beea36513f66e9cbe43 [id=e12703cc30a94230b4c5f0fc21988359]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.541512 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1938e67acaf4ab19fcbed13e1f1bfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.543136 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf
I20250629 01:51:40.543547 450 catalog_manager.cc:6652] ReplaceTablet: tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf deleted and replaced by tablet a2562570cb0846e19ebc97bdb43bd70d
I20250629 01:51:40.543803 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3abcbdb38a38484782a28269e7b69beb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.544253 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3abcbdb38a38484782a28269e7b69beb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.545709 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]). Expected version 2 got 0
I20250629 01:51:40.545905 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]). Expected version 2 got 0
I20250629 01:51:40.546099 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763]). Expected version 2 got 0
I20250629 01:51:40.546344 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_3194edfea46f4f9e8e989565b3438162 [id=ac80888442774af495a53bba44dfa763] alter complete (version 2)
I20250629 01:51:40.546962 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7768d2d7c9814ba48196e54d12a630a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.547560 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7768d2d7c9814ba48196e54d12a630a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:40.548126 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d44151acb0d4bf4a630c086b1ac8ff7 consensus state has no leader
I20250629 01:51:40.549140 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf with delete_type TABLET_DATA_DELETED (replaced by tablet a2562570cb0846e19ebc97bdb43bd70d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.550669 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:40.551201 1157 tablet_replica.cc:331] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.551384 1157 raft_consensus.cc:2241] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.551524 1157 raft_consensus.cc:2270] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.551993 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]). Expected version 2 got 0
I20250629 01:51:40.552470 1157 ts_tablet_manager.cc:1905] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.552495 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]). Expected version 1 got 0
I20250629 01:51:40.552685 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:40.552848 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6]). Expected version 2 got 0
I20250629 01:51:40.554939 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d237b9f962024ecb967f86a5730a5039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.555148 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d46683318dc4c04a617a99f3e620734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.538851 1154 tablet_bootstrap.cc:654] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.557446 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.557705 1158 tablet_replica.cc:331] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.557919 1158 raft_consensus.cc:2241] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.558075 1158 raft_consensus.cc:2270] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.560026 1158 ts_tablet_manager.cc:1905] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.561137 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c586a09ba29642f394d315d9b8ae758c" } modify_external_catalogs: true
I20250629 01:51:40.561316 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c586a09ba29642f394d315d9b8ae758c" } modify_external_catalogs: true
I20250629 01:51:40.561787 1154 tablet_bootstrap.cc:492] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.561895 1154 ts_tablet_manager.cc:1397] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20250629 01:51:40.562163 1174 raft_consensus.cc:491] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:40.562320 1174 raft_consensus.cc:513] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.562784 1174 leader_election.cc:290] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:40.563597 1154 raft_consensus.cc:357] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.563741 1154 raft_consensus.cc:383] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.563851 1154 raft_consensus.cc:738] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.564060 1154 consensus_queue.cc:260] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.564471 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 05905638c6e241a095438f277f53adab
I20250629 01:51:40.565227 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc6f34d6b31849e190ebfaf1a273d29c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:40.565282 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bd5b835ff8624ebf8956eeb965ce5950
I20250629 01:51:40.565392 916 raft_consensus.cc:2466] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:40.565433 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1938e67acaf4ab19fcbed13e1f1bfc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.565651 452 catalog_manager.cc:6652] ReplaceTablet: tablet bd5b835ff8624ebf8956eeb965ce5950 deleted and replaced by tablet a2a9664bb7a946c48cf79a09955ecad5
W20250629 01:51:40.566099 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb359994b20748518a676045ac6f0df8 consensus state has no leader
I20250629 01:51:40.566061 988 leader_election.cc:304] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.566424 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c45882f8dce24c1bafebd9271c4bbf3d
I20250629 01:51:40.566507 1174 raft_consensus.cc:2802] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.566582 1174 raft_consensus.cc:491] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.566664 1174 raft_consensus.cc:3058] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.568867 892 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 with delete_type TABLET_DATA_DELETED (replaced by tablet a2a9664bb7a946c48cf79a09955ecad5) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.569159 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf with delete_type TABLET_DATA_DELETED (replaced by tablet a2562570cb0846e19ebc97bdb43bd70d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.569133 1174 raft_consensus.cc:513] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.569159 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 05905638c6e241a095438f277f53adab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.569476 1156 tablet_replica.cc:331] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.569662 1156 raft_consensus.cc:2241] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.569810 1156 raft_consensus.cc:2270] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.570008 752 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 with delete_type TABLET_DATA_DELETED (replaced by tablet a2a9664bb7a946c48cf79a09955ecad5) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.570519 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 with delete_type TABLET_DATA_DELETED (replaced by tablet a2a9664bb7a946c48cf79a09955ecad5) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.570581 1158 ts_tablet_manager.cc:1918] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.570708 1158 log.cc:1199] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/267d9a050fa2414d87a04ba3c91dea03
I20250629 01:51:40.570838 1156 ts_tablet_manager.cc:1905] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.571018 1158 ts_tablet_manager.cc:1939] T 267d9a050fa2414d87a04ba3c91dea03 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.571949 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 05905638c6e241a095438f277f53adab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.572176 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1185e85345664dfd92c134c5552e7d48" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:40.572357 777 raft_consensus.cc:2466] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:40.572702 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 267d9a050fa2414d87a04ba3c91dea03 (table default.Table_8c79e92b6cfe429088a6b6662f46412c [id=2819c6ba47aa436a9ddd9a72f956c96e]) successfully deleted
W20250629 01:51:40.572896 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet bd5b835ff8624ebf8956eeb965ce5950 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b835ff8624ebf8956eeb965ce5950 already in progress: creating tablet
I20250629 01:51:40.573146 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 05905638c6e241a095438f277f53adab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.573522 1154 ts_tablet_manager.cc:1428] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250629 01:51:40.573786 1154 tablet_bootstrap.cc:492] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.574111 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]). Expected version 1 got 0
I20250629 01:51:40.575230 1154 tablet_bootstrap.cc:654] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.575518 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc6f34d6b31849e190ebfaf1a273d29c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:40.575691 916 raft_consensus.cc:3058] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:40.576323 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 05905638c6e241a095438f277f53adab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05905638c6e241a095438f277f53adab already in progress: creating tablet
I20250629 01:51:40.576705 1159 tablet_bootstrap.cc:492] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.576869 1159 ts_tablet_manager.cc:1397] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.019s
I20250629 01:51:40.577529 1157 ts_tablet_manager.cc:1918] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.577575 1159 raft_consensus.cc:357] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:40.577754 1159 raft_consensus.cc:383] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.577826 1159 raft_consensus.cc:738] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.578037 850 leader_election.cc:304] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:40.578154 916 raft_consensus.cc:2466] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:40.578351 757 tablet_service.cc:1468] Processing CreateTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 (DEFAULT_TABLE table=default.Table_358f4795c7e244f2b72efe50dacd310f [id=00b7c47fc4684174b514feaab3b7d411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.578635 1227 raft_consensus.cc:2802] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.578688 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829c64e2c71347f4af2c8870cdd8af31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.578728 1227 raft_consensus.cc:491] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.578795 1227 raft_consensus.cc:3058] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.579077 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 7748846a3df14ffa801321ccde86e35f (DEFAULT_TABLE table=default.Table_9edf5b31c7b24325930accee8cb627c3 [id=79a016a743f641158a9240b362c274c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.579381 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7748846a3df14ffa801321ccde86e35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.581158 1227 raft_consensus.cc:513] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.581679 1227 leader_election.cc:290] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.582232 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1185e85345664dfd92c134c5552e7d48" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:40.582264 889 tablet_service.cc:1468] Processing CreateTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 (DEFAULT_TABLE table=default.Table_358f4795c7e244f2b72efe50dacd310f [id=00b7c47fc4684174b514feaab3b7d411]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.582590 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829c64e2c71347f4af2c8870cdd8af31. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:40.582648 850 leader_election.cc:343] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:40.578181 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc6f34d6b31849e190ebfaf1a273d29c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:40.582187 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1185e85345664dfd92c134c5552e7d48" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:40.582940 776 raft_consensus.cc:3058] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.582959 775 raft_consensus.cc:3058] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.584332 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc6f34d6b31849e190ebfaf1a273d29c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:40.584755 988 leader_election.cc:304] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.585569 775 raft_consensus.cc:2466] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:40.585846 892 tablet_service.cc:1468] Processing CreateTablet for tablet 7748846a3df14ffa801321ccde86e35f (DEFAULT_TABLE table=default.Table_9edf5b31c7b24325930accee8cb627c3 [id=79a016a743f641158a9240b362c274c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.586118 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7748846a3df14ffa801321ccde86e35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.586315 752 tablet_service.cc:1468] Processing CreateTablet for tablet 97439c6a6e844610b55b899966ae5b67 (DEFAULT_TABLE table=default.Table_aa33973948ab4d68b062deca5d60a668 [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.586453 776 raft_consensus.cc:2466] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:40.586665 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97439c6a6e844610b55b899966ae5b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.590114 1174 leader_election.cc:290] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:40.590103 850 leader_election.cc:304] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:40.590265 1174 raft_consensus.cc:2802] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.590881 1227 raft_consensus.cc:2802] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.591352 1227 raft_consensus.cc:695] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:40.591642 1227 consensus_queue.cc:237] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.592501 1174 raft_consensus.cc:695] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:40.595317 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]). Expected version 1 got 0
W20250629 01:51:40.597493 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2562570cb0846e19ebc97bdb43bd70d has no consensus state
I20250629 01:51:40.601289 1159 consensus_queue.cc:260] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:40.604522 892 tablet_service.cc:1468] Processing CreateTablet for tablet 97439c6a6e844610b55b899966ae5b67 (DEFAULT_TABLE table=default.Table_aa33973948ab4d68b062deca5d60a668 [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.604869 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97439c6a6e844610b55b899966ae5b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.605335 1174 consensus_queue.cc:237] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.608002 1159 ts_tablet_manager.cc:1428] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250629 01:51:40.609643 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 (DEFAULT_TABLE table=default.Table_534d46b1a5ee4435844d4dd7c6626d1f [id=ff73871a84f745e383b03505bf30192d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.610025 1159 tablet_bootstrap.cc:492] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.612188 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 (DEFAULT_TABLE table=default.Table_534d46b1a5ee4435844d4dd7c6626d1f [id=ff73871a84f745e383b03505bf30192d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.612511 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6128463cd85d4f0cbfe21b0e5bd0e470. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.612872 1159 tablet_bootstrap.cc:654] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.612883 754 tablet_service.cc:1468] Processing CreateTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 (DEFAULT_TABLE table=default.Table_2ce50be1c3444c059ed91faaf9cd5854 [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.615144 896 tablet_service.cc:1468] Processing CreateTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 (DEFAULT_TABLE table=default.Table_2ce50be1c3444c059ed91faaf9cd5854 [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.615491 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe14d1d54db74a99b05cf8b1a60827a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.620018 1156 ts_tablet_manager.cc:1918] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.622843 750 tablet_service.cc:1515] Processing DeleteTablet for tablet d237b9f962024ecb967f86a5730a5039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.623109 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6128463cd85d4f0cbfe21b0e5bd0e470. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.626381 1164 raft_consensus.cc:491] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:40.626564 1164 raft_consensus.cc:513] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.627233 1164 leader_election.cc:290] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.628314 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe14d1d54db74a99b05cf8b1a60827a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.628569 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d46683318dc4c04a617a99f3e620734" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:40.628736 916 raft_consensus.cc:2466] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:40.632436 712 leader_election.cc:304] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:40.633188 1025 tablet_service.cc:1468] Processing CreateTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 (DEFAULT_TABLE table=default.Table_2ce50be1c3444c059ed91faaf9cd5854 [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.633659 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe14d1d54db74a99b05cf8b1a60827a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.636503 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 (DEFAULT_TABLE table=default.Table_534d46b1a5ee4435844d4dd7c6626d1f [id=ff73871a84f745e383b03505bf30192d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.636884 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6128463cd85d4f0cbfe21b0e5bd0e470. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.640293 1164 raft_consensus.cc:2802] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.640420 1164 raft_consensus.cc:491] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.640509 1164 raft_consensus.cc:3058] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.643687 1164 raft_consensus.cc:513] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:40.644582 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 59853us
I20250629 01:51:40.644842 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_38d08e8c1c0d4beea36513f66e9cbe43" } modify_external_catalogs: true
I20250629 01:51:40.645052 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_38d08e8c1c0d4beea36513f66e9cbe43" } modify_external_catalogs: true
I20250629 01:51:40.646474 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_534d46b1a5ee4435844d4dd7c6626d1f" } modify_external_catalogs: true
I20250629 01:51:40.646637 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_534d46b1a5ee4435844d4dd7c6626d1f" } modify_external_catalogs: true
I20250629 01:51:40.646771 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d46683318dc4c04a617a99f3e620734" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:40.647307 710 leader_election.cc:343] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:40.647722 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]). Expected version 1 got 0
I20250629 01:51:40.647915 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:40.648165 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6]). Expected version 2 got 0
I20250629 01:51:40.651846 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet d237b9f962024ecb967f86a5730a5039 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.653143 1158 tablet_replica.cc:331] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.653353 1158 raft_consensus.cc:2241] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.653527 1158 raft_consensus.cc:2270] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.653821 448 catalog_manager.cc:5582] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:40.654250 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]). Expected version 2 got 0
I20250629 01:51:40.654315 1158 ts_tablet_manager.cc:1905] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.661118 1158 ts_tablet_manager.cc:1918] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.661836 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 97e03c52aeb4432cab002abdae5109ba from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.662564 1158 log.cc:1199] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d237b9f962024ecb967f86a5730a5039
I20250629 01:51:40.663522 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d46683318dc4c04a617a99f3e620734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.664588 1164 leader_election.cc:290] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.664608 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d46683318dc4c04a617a99f3e620734" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:40.664776 916 raft_consensus.cc:3058] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.667155 916 raft_consensus.cc:2466] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:40.667464 712 leader_election.cc:304] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:40.668516 1158 ts_tablet_manager.cc:1939] T d237b9f962024ecb967f86a5730a5039 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.669497 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d237b9f962024ecb967f86a5730a5039 (table default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]) successfully deleted
I20250629 01:51:40.662825 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf with delete_type TABLET_DATA_DELETED (replaced by tablet a2562570cb0846e19ebc97bdb43bd70d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.665025 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d46683318dc4c04a617a99f3e620734" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
W20250629 01:51:40.670158 710 leader_election.cc:343] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:40.670358 1164 raft_consensus.cc:2802] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.671602 1164 raft_consensus.cc:695] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:40.671847 1164 consensus_queue.cc:237] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.673035 1156 log.cc:1199] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/01eaf627f1dd41c4b5ad6c0a46e2e7cf
I20250629 01:51:40.673460 1156 ts_tablet_manager.cc:1939] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.577662 1157 log.cc:1199] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/01eaf627f1dd41c4b5ad6c0a46e2e7cf
I20250629 01:51:40.676252 1157 ts_tablet_manager.cc:1939] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.676707 1157 tablet_replica.cc:331] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.676908 1157 raft_consensus.cc:2241] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.677076 1157 raft_consensus.cc:2270] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.680657 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_737eb1cf8b184a14b253c25d114a14b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.680764 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8eedc26a9ab14d5c8dd30631e6ec1fe8" } new_table_name: "default.Table_bba7ed21c41c46ca8f68ce01c5225b1e" modify_external_catalogs: true
I20250629 01:51:40.681056 1227 raft_consensus.cc:491] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:40.681166 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_737eb1cf8b184a14b253c25d114a14b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.681195 1227 raft_consensus.cc:513] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.681219 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1e32538a1c65431dbe7e57032b44c77e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.681681 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e32538a1c65431dbe7e57032b44c77e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.681738 1227 leader_election.cc:290] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.681846 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ebdea537fafa4ed88bb3006569ce22c9" } new_table_name: "default.Table_36ee44a1ebdd4553a043949dd8cdb50e" modify_external_catalogs: true
I20250629 01:51:40.682758 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c0b48fb812c9461092e2d5e0f18b38c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.683140 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0b48fb812c9461092e2d5e0f18b38c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.683899 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3c896a37d9749c5a46bfc7cf7c87911" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:40.684073 776 raft_consensus.cc:2466] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:40.684286 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3c896a37d9749c5a46bfc7cf7c87911" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:40.684476 1052 raft_consensus.cc:2466] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:40.685142 1157 ts_tablet_manager.cc:1905] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:40.685737 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf already in progress: creating tablet
I20250629 01:51:40.686097 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (table default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce]) successfully deleted
I20250629 01:51:40.685177 850 leader_election.cc:304] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.686678 1227 raft_consensus.cc:2802] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.686766 1227 raft_consensus.cc:491] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.686857 1227 raft_consensus.cc:3058] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.688050 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (table default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce]) successfully deleted
I20250629 01:51:40.688503 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.688827 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 97439c6a6e844610b55b899966ae5b67 (DEFAULT_TABLE table=default.Table_aa33973948ab4d68b062deca5d60a668 [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.689113 1227 raft_consensus.cc:513] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.689641 1227 leader_election.cc:290] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.690096 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fe6e1e274ae04e02bc6d154720f71759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.690760 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe6e1e274ae04e02bc6d154720f71759 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.697376 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d4bd19f9578f4fa19a6daef427e4a6c8
I20250629 01:51:40.697911 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 97e03c52aeb4432cab002abdae5109ba from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.698045 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:32789
--webserver_interface=127.0.104.60
--webserver_port=34121
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250629 01:51:40.698685 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3c896a37d9749c5a46bfc7cf7c87911" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:40.698832 776 raft_consensus.cc:3058] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.698783 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3c896a37d9749c5a46bfc7cf7c87911" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:40.698951 1052 raft_consensus.cc:3058] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.701112 776 raft_consensus.cc:2466] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:40.702098 1154 tablet_bootstrap.cc:492] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.702258 1154 ts_tablet_manager.cc:1397] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.128s user 0.013s sys 0.023s
I20250629 01:51:40.702263 1052 raft_consensus.cc:2466] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:40.702986 1154 raft_consensus.cc:357] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.703135 1154 raft_consensus.cc:383] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.703191 1154 raft_consensus.cc:738] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.703414 1154 consensus_queue.cc:260] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.704123 1154 ts_tablet_manager.cc:1428] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:40.704406 1154 tablet_bootstrap.cc:492] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.705431 850 leader_election.cc:304] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.705699 1227 raft_consensus.cc:2802] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.705825 1227 raft_consensus.cc:695] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:40.706056 1227 consensus_queue.cc:237] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.708266 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a1938e67acaf4ab19fcbed13e1f1bfc6
W20250629 01:51:40.708588 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a72c6040421642578fac307ea9914ef6 consensus state has no leader
I20250629 01:51:40.708823 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_502b190b8a7046819a8d71733a10c839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.708884 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_adc0a47bc8504a73aa8b9ede02097dd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.709199 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_502b190b8a7046819a8d71733a10c839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:40.709307 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adc0a47bc8504a73aa8b9ede02097dd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.710767 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6128463cd85d4f0cbfe21b0e5bd0e470
I20250629 01:51:40.717654 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.718439 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:40.718801 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1938e67acaf4ab19fcbed13e1f1bfc6 already in progress: creating tablet
I20250629 01:51:40.719051 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.719431 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2c04531a50c54791a64d6537b670d54f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.719646 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6128463cd85d4f0cbfe21b0e5bd0e470 already in progress: creating tablet
W20250629 01:51:40.719945 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c04531a50c54791a64d6537b670d54f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.722024 891 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.722277 892 tablet_service.cc:1515] Processing DeleteTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.722373 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.723696 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:40.724138 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d4bd19f9578f4fa19a6daef427e4a6c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4bd19f9578f4fa19a6daef427e4a6c8 already in progress: creating tablet
I20250629 01:51:40.724236 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.726302 753 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.726617 751 tablet_service.cc:1468] Processing CreateTablet for tablet 7748846a3df14ffa801321ccde86e35f (DEFAULT_TABLE table=default.Table_9edf5b31c7b24325930accee8cb627c3 [id=79a016a743f641158a9240b362c274c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.726962 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7748846a3df14ffa801321ccde86e35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.729063 1156 tablet_replica.cc:331] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.729095 453 master_service.cc:948] ReplaceTablet: received request to replace tablet bd5b835ff8624ebf8956eeb965ce5950 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.729225 1156 raft_consensus.cc:2241] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:40.729497 1156 pending_rounds.cc:70] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:40.729578 1156 pending_rounds.cc:77] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759144378351616 op_type: NO_OP noop_request { }
I20250629 01:51:40.729704 1156 raft_consensus.cc:2887] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:40.729807 1156 raft_consensus.cc:2270] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.730047 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97439c6a6e844610b55b899966ae5b67. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:40.730762 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f60c48c924d49a186a8c81d154bdc54 has no consensus state
I20250629 01:51:40.731731 1154 tablet_bootstrap.cc:654] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:40.735460 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 05905638c6e241a095438f277f53adab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05905638c6e241a095438f277f53adab already in progress: creating tablet
W20250629 01:51:40.735707 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet bd5b835ff8624ebf8956eeb965ce5950 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b835ff8624ebf8956eeb965ce5950 already in progress: creating tablet
I20250629 01:51:40.743876 1154 tablet_bootstrap.cc:492] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.744028 1154 ts_tablet_manager.cc:1397] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.006s
I20250629 01:51:40.744791 1154 raft_consensus.cc:357] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.744985 1154 raft_consensus.cc:383] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.745095 1154 raft_consensus.cc:738] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.745339 1154 consensus_queue.cc:260] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.745903 1154 ts_tablet_manager.cc:1428] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:40.746167 1154 tablet_bootstrap.cc:492] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.747707 1154 tablet_bootstrap.cc:654] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.749600 1157 ts_tablet_manager.cc:1918] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.749727 1157 log.cc:1199] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d237b9f962024ecb967f86a5730a5039
I20250629 01:51:40.742538 1156 ts_tablet_manager.cc:1905] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.750101 1157 ts_tablet_manager.cc:1939] T d237b9f962024ecb967f86a5730a5039 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.750627 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d237b9f962024ecb967f86a5730a5039 (table default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]) successfully deleted
I20250629 01:51:40.753378 446 master_service.cc:948] ReplaceTablet: received request to replace tablet a229b4f9d19147a5990f799d1dc0e5d7 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.758594 1157 tablet_replica.cc:331] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.758702 1154 tablet_bootstrap.cc:492] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.758791 1157 raft_consensus.cc:2241] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.758899 1154 ts_tablet_manager.cc:1397] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20250629 01:51:40.758939 1157 raft_consensus.cc:2270] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.759660 1155 tablet_bootstrap.cc:492] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.759792 1155 ts_tablet_manager.cc:1397] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.259s user 0.042s sys 0.120s
I20250629 01:51:40.759811 1154 raft_consensus.cc:357] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.760216 1154 raft_consensus.cc:383] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.760438 1154 raft_consensus.cc:738] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.760545 1155 raft_consensus.cc:357] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.760774 1155 raft_consensus.cc:383] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.760867 1155 raft_consensus.cc:738] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.761047 1154 consensus_queue.cc:260] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.761672 1157 ts_tablet_manager.cc:1905] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.761659 1155 consensus_queue.cc:260] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.761998 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_16db51abc1404d7fa67f1c00a70f9166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.762169 1154 ts_tablet_manager.cc:1428] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:51:40.762277 1155 ts_tablet_manager.cc:1428] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:40.762419 1154 tablet_bootstrap.cc:492] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:40.762491 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16db51abc1404d7fa67f1c00a70f9166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.762531 1155 tablet_bootstrap.cc:492] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.763877 1155 tablet_bootstrap.cc:654] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.767561 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7abb332bd3ed4d7299fc9f6959013250" } new_table_name: "default.Table_7b3fdda8530a4fefbb38859d44f183af" modify_external_catalogs: true
I20250629 01:51:40.768741 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_07d0ae14810e4c31b91252f2b672ef5d" } modify_external_catalogs: true
I20250629 01:51:40.768911 1154 tablet_bootstrap.cc:654] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.768901 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_07d0ae14810e4c31b91252f2b672ef5d" } modify_external_catalogs: true
I20250629 01:51:40.771484 1157 ts_tablet_manager.cc:1918] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.771597 1157 log.cc:1199] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0d46683318dc4c04a617a99f3e620734
I20250629 01:51:40.771852 1157 ts_tablet_manager.cc:1939] T 0d46683318dc4c04a617a99f3e620734 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.778210 1154 tablet_bootstrap.cc:492] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.778491 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bd59d9e2715745828993ae11d0dd56d0" } modify_external_catalogs: true
I20250629 01:51:40.778652 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_36ee44a1ebdd4553a043949dd8cdb50e" } new_table_name: "default.Table_376190835c644f499104ffbbd085ac10" modify_external_catalogs: true
I20250629 01:51:40.778695 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bd59d9e2715745828993ae11d0dd56d0" } modify_external_catalogs: true
I20250629 01:51:40.779042 1154 ts_tablet_manager.cc:1397] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.002s
I20250629 01:51:40.779659 1154 raft_consensus.cc:357] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.779808 1154 raft_consensus.cc:383] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.779891 1154 raft_consensus.cc:738] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.780140 1154 consensus_queue.cc:260] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.780689 1154 ts_tablet_manager.cc:1428] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:40.780900 1154 tablet_bootstrap.cc:492] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:40.782199 1154 tablet_bootstrap.cc:654] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.784055 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 648fd1ca8f514ea3ba4554452b19933f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.786999 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0d46683318dc4c04a617a99f3e620734 (table default.Table_45d1774983d248aeab1a0353a2d1ece2 [id=f7df5c11e9a84f61b749ebf60a50e6b8]) successfully deleted
W20250629 01:51:40.787794 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet bd5b835ff8624ebf8956eeb965ce5950 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd5b835ff8624ebf8956eeb965ce5950 already in progress: creating tablet
W20250629 01:51:40.788054 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 05905638c6e241a095438f277f53adab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05905638c6e241a095438f277f53adab already in progress: creating tablet
W20250629 01:51:40.788352 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d4bd19f9578f4fa19a6daef427e4a6c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4bd19f9578f4fa19a6daef427e4a6c8 already in progress: creating tablet
W20250629 01:51:40.788570 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1938e67acaf4ab19fcbed13e1f1bfc6 already in progress: creating tablet
W20250629 01:51:40.788803 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6128463cd85d4f0cbfe21b0e5bd0e470 already in progress: creating tablet
I20250629 01:51:40.791642 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.794467 1154 tablet_bootstrap.cc:492] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.794625 1154 ts_tablet_manager.cc:1397] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20250629 01:51:40.795296 1154 raft_consensus.cc:357] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.796484 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]). Expected version 2 got 0
I20250629 01:51:40.796665 455 catalog_manager.cc:5582] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:40.796990 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]). Expected version 1 got 0
I20250629 01:51:40.798349 451 catalog_manager.cc:5582] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:40.798683 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:40.799959 1159 tablet_bootstrap.cc:492] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.800074 1159 ts_tablet_manager.cc:1397] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.190s user 0.026s sys 0.032s
I20250629 01:51:40.800168 1154 raft_consensus.cc:383] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.800248 1154 raft_consensus.cc:738] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.800469 1154 consensus_queue.cc:260] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.800791 1159 raft_consensus.cc:357] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.800938 1159 raft_consensus.cc:383] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.800994 1154 ts_tablet_manager.cc:1428] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:40.801014 1159 raft_consensus.cc:738] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.801214 1159 consensus_queue.cc:260] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:40.801607 1159 ts_tablet_manager.cc:1428] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:40.801857 1159 tablet_bootstrap.cc:492] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:40.803010 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f60c48c924d49a186a8c81d154bdc54 consensus state has no leader
W20250629 01:51:40.804396 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd6013bb40c248f29f40eab2c3008f6b consensus state has no leader
I20250629 01:51:40.804497 1159 tablet_bootstrap.cc:654] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.815703 891 tablet_service.cc:1468] Processing CreateTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 (DEFAULT_TABLE table=default.Table_97eb293848a64879af553f06d6660c31 [id=ab297380d8bd4ffa8db5cfbff08519ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.816152 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e61d9576f8414ebf0074d57cdfb1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.823805 1159 tablet_bootstrap.cc:492] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.823938 1159 ts_tablet_manager.cc:1397] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20250629 01:51:40.824590 1159 raft_consensus.cc:357] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.824776 1159 raft_consensus.cc:383] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.824923 1154 tablet_bootstrap.cc:492] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:40.827378 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet dd6013bb40c248f29f40eab2c3008f6b (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dd6013bb40c248f29f40eab2c3008f6b consensus state has no leader
I20250629 01:51:40.827620 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5e9f6de505a34afbae48a373b4b2a0cf
I20250629 01:51:40.827912 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3194edfea46f4f9e8e989565b3438162" } new_table_name: "default.Table_d1f1005233b74726b856dba70c0a4b8b" modify_external_catalogs: true
I20250629 01:51:40.828461 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 648fd1ca8f514ea3ba4554452b19933f
I20250629 01:51:40.828908 452 catalog_manager.cc:6652] ReplaceTablet: tablet 648fd1ca8f514ea3ba4554452b19933f deleted and replaced by tablet 0b7a640451934bff8672cd88dc86c529
I20250629 01:51:40.829429 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1185e85345664dfd92c134c5552e7d48
I20250629 01:51:40.831332 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.831537 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 648fd1ca8f514ea3ba4554452b19933f with delete_type TABLET_DATA_DELETED (replaced by tablet 0b7a640451934bff8672cd88dc86c529) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.831660 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 648fd1ca8f514ea3ba4554452b19933f with delete_type TABLET_DATA_DELETED (replaced by tablet 0b7a640451934bff8672cd88dc86c529) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.831698 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 1185e85345664dfd92c134c5552e7d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.831894 1159 raft_consensus.cc:738] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.832124 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 1185e85345664dfd92c134c5552e7d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.832208 1159 consensus_queue.cc:260] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.832336 1157 tablet_replica.cc:331] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.832490 1157 raft_consensus.cc:2241] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:40.832859 1157 pending_rounds.cc:70] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Trying to abort 1 pending ops.
I20250629 01:51:40.833022 1157 pending_rounds.cc:77] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759144828010496 op_type: NO_OP noop_request { }
I20250629 01:51:40.833137 1157 raft_consensus.cc:2887] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:40.833232 1157 raft_consensus.cc:2270] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:40.833760 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 648fd1ca8f514ea3ba4554452b19933f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648fd1ca8f514ea3ba4554452b19933f already in progress: creating tablet
I20250629 01:51:40.833978 1159 ts_tablet_manager.cc:1428] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:40.834228 1159 tablet_bootstrap.cc:492] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.831660 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.835901 1159 tablet_bootstrap.cc:654] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.836799 1157 ts_tablet_manager.cc:1905] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.837983 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 (DEFAULT_TABLE table=default.Table_97eb293848a64879af553f06d6660c31 [id=ab297380d8bd4ffa8db5cfbff08519ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.844723 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 648fd1ca8f514ea3ba4554452b19933f with delete_type TABLET_DATA_DELETED (replaced by tablet 0b7a640451934bff8672cd88dc86c529) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.845395 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:40.846550 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 648fd1ca8f514ea3ba4554452b19933f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648fd1ca8f514ea3ba4554452b19933f already in progress: creating tablet
W20250629 01:51:40.847095 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5e9f6de505a34afbae48a373b4b2a0cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e9f6de505a34afbae48a373b4b2a0cf already in progress: creating tablet
I20250629 01:51:40.844558 1156 ts_tablet_manager.cc:1918] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:40.847496 1156 log.cc:1199] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d237b9f962024ecb967f86a5730a5039
I20250629 01:51:40.847589 1154 tablet_bootstrap.cc:654] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.848685 1156 ts_tablet_manager.cc:1939] T d237b9f962024ecb967f86a5730a5039 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.848757 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 1185e85345664dfd92c134c5552e7d48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.848320 1155 tablet_bootstrap.cc:492] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:40.848943 1158 tablet_replica.cc:331] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.849042 1155 ts_tablet_manager.cc:1397] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.087s user 0.020s sys 0.023s
I20250629 01:51:40.849102 1158 raft_consensus.cc:2241] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.849242 1158 raft_consensus.cc:2270] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.849817 1158 ts_tablet_manager.cc:1905] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.850721 1155 raft_consensus.cc:357] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.850991 1155 raft_consensus.cc:383] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.851322 1155 raft_consensus.cc:738] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.851610 1155 consensus_queue.cc:260] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.851236 1156 tablet_replica.cc:331] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.851975 1156 raft_consensus.cc:2241] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:40.854157 1156 pending_rounds.cc:70] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:40.854578 1156 pending_rounds.cc:77] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759145154359296 op_type: NO_OP noop_request { }
I20250629 01:51:40.854728 1156 raft_consensus.cc:2887] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:40.854858 1156 raft_consensus.cc:2270] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.854507 1155 ts_tablet_manager.cc:1428] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:40.855290 1158 ts_tablet_manager.cc:1918] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.855406 1158 log.cc:1199] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/1185e85345664dfd92c134c5552e7d48
I20250629 01:51:40.855700 1158 ts_tablet_manager.cc:1939] T 1185e85345664dfd92c134c5552e7d48 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:40.856130 1275 log.cc:927] Time spent T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20250629 01:51:40.856173 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d237b9f962024ecb967f86a5730a5039 (table default.Table_ade22ca69a16455b9d8ded9bb0a060dc [id=c22f2e9e8d084f2089def95c26c2b2de]) successfully deleted
I20250629 01:51:40.857096 1155 tablet_bootstrap.cc:492] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:40.858459 1155 tablet_bootstrap.cc:654] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.859181 1159 tablet_bootstrap.cc:492] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:40.859308 1159 ts_tablet_manager.cc:1397] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20250629 01:51:40.859920 1159 raft_consensus.cc:357] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.860021 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e61d9576f8414ebf0074d57cdfb1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.860059 1159 raft_consensus.cc:383] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.860152 1159 raft_consensus.cc:738] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.860342 1159 consensus_queue.cc:260] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.860954 1159 ts_tablet_manager.cc:1428] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:40.861223 1159 tablet_bootstrap.cc:492] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:40.861713 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]) successfully deleted
I20250629 01:51:40.862391 1157 ts_tablet_manager.cc:1918] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:40.862514 1157 log.cc:1199] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/1185e85345664dfd92c134c5552e7d48
I20250629 01:51:40.862880 1157 ts_tablet_manager.cc:1939] T 1185e85345664dfd92c134c5552e7d48 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:40.864023 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]) successfully deleted
I20250629 01:51:40.865971 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.866642 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b710052005c4a2aaade1dc60409aef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.867067 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.869946 751 tablet_service.cc:1468] Processing CreateTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 (DEFAULT_TABLE table=default.Table_97eb293848a64879af553f06d6660c31 [id=ab297380d8bd4ffa8db5cfbff08519ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.870275 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e61d9576f8414ebf0074d57cdfb1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.871773 450 master_service.cc:948] ReplaceTablet: received request to replace tablet dc6f34d6b31849e190ebfaf1a273d29c from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.871937 1159 tablet_bootstrap.cc:654] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:40.872514 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 267d9a050fa2414d87a04ba3c91dea03 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.873317 1156 ts_tablet_manager.cc:1905] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.874857 1164 raft_consensus.cc:491] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:40.874899 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5e9f6de505a34afbae48a373b4b2a0cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e9f6de505a34afbae48a373b4b2a0cf already in progress: creating tablet
I20250629 01:51:40.874996 1164 raft_consensus.cc:513] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.875303 1157 tablet_replica.cc:331] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:40.875454 1157 raft_consensus.cc:2241] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.875523 1164 leader_election.cc:290] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.875617 1157 raft_consensus.cc:2270] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.875996 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57f7f71ab5e54d249141c89d9e806111" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:40.876161 916 raft_consensus.cc:2466] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:40.876686 712 leader_election.cc:304] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:40.876726 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57f7f71ab5e54d249141c89d9e806111" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:40.876871 1052 raft_consensus.cc:2466] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:40.877025 1164 raft_consensus.cc:2802] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.877136 1164 raft_consensus.cc:491] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.877241 1164 raft_consensus.cc:3058] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.879352 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6]). Expected version 2 got 0
I20250629 01:51:40.881578 1157 ts_tablet_manager.cc:1905] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.882030 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_99a319fbbffa4df69d714b2a48777387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.882483 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99a319fbbffa4df69d714b2a48777387 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.885278 1164 raft_consensus.cc:513] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.885921 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet dc6f34d6b31849e190ebfaf1a273d29c
I20250629 01:51:40.886296 450 catalog_manager.cc:6652] ReplaceTablet: tablet dc6f34d6b31849e190ebfaf1a273d29c deleted and replaced by tablet bedbb041e4094d8399de625badd971bb
I20250629 01:51:40.886211 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57f7f71ab5e54d249141c89d9e806111" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:40.886386 916 raft_consensus.cc:3058] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.887969 1156 ts_tablet_manager.cc:1918] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:40.888067 1156 log.cc:1199] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0d46683318dc4c04a617a99f3e620734
I20250629 01:51:40.889223 916 raft_consensus.cc:2466] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:40.890209 712 leader_election.cc:304] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:40.890532 1263 raft_consensus.cc:2802] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.890664 1263 raft_consensus.cc:695] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:40.891009 1263 consensus_queue.cc:237] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.893074 1164 leader_election.cc:290] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:40.893138 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57f7f71ab5e54d249141c89d9e806111" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:40.893306 1052 raft_consensus.cc:3058] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.895908 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf with delete_type TABLET_DATA_DELETED (replaced by tablet a2562570cb0846e19ebc97bdb43bd70d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.896222 1156 ts_tablet_manager.cc:1939] T 0d46683318dc4c04a617a99f3e620734 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.896054 1052 raft_consensus.cc:2466] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:40.897838 1156 tablet_replica.cc:331] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.898213 1156 raft_consensus.cc:2241] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.898248 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b3395b9c2d04a6f8a227630087d85d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.898339 892 tablet_service.cc:1515] Processing DeleteTablet for tablet dc6f34d6b31849e190ebfaf1a273d29c with delete_type TABLET_DATA_DELETED (replaced by tablet bedbb041e4094d8399de625badd971bb) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.898581 1158 tablet_replica.cc:331] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:40.898743 1158 raft_consensus.cc:2241] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.898569 1156 raft_consensus.cc:2270] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.898905 1158 raft_consensus.cc:2270] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.899422 1156 ts_tablet_manager.cc:1905] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.899430 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet ee70cbe4d80e41b99234f1e1b001434b with delete_type TABLET_DATA_DELETED (replaced by tablet 7ae96f2da36744198e3d38cb07bb47a0) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.899904 1158 ts_tablet_manager.cc:1905] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.900632 749 tablet_service.cc:1515] Processing DeleteTablet for tablet dc6f34d6b31849e190ebfaf1a273d29c with delete_type TABLET_DATA_DELETED (replaced by tablet bedbb041e4094d8399de625badd971bb) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.901353 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet dc6f34d6b31849e190ebfaf1a273d29c with delete_type TABLET_DATA_DELETED (replaced by tablet bedbb041e4094d8399de625badd971bb) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.901872 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0d46683318dc4c04a617a99f3e620734 (table default.Table_45d1774983d248aeab1a0353a2d1ece2 [id=f7df5c11e9a84f61b749ebf60a50e6b8]) successfully deleted
I20250629 01:51:40.905069 1174 raft_consensus.cc:491] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:40.905243 1174 raft_consensus.cc:513] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.905766 1174 leader_election.cc:290] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:40.907467 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d7fba939fd154c0ba4dcea2e652fc6aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.906121 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3395b9c2d04a6f8a227630087d85d8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:40.907778 916 raft_consensus.cc:2466] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
W20250629 01:51:40.907991 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7fba939fd154c0ba4dcea2e652fc6aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.908155 988 leader_election.cc:304] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.908392 1174 raft_consensus.cc:2802] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:40.908479 1174 raft_consensus.cc:491] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:40.908555 1174 raft_consensus.cc:3058] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.907573 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_376190835c644f499104ffbbd085ac10" } modify_external_catalogs: true
I20250629 01:51:40.909199 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_376190835c644f499104ffbbd085ac10" } modify_external_catalogs: true
I20250629 01:51:40.910831 1174 raft_consensus.cc:513] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.911624 1174 leader_election.cc:290] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:40.911689 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3395b9c2d04a6f8a227630087d85d8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:40.911862 916 raft_consensus.cc:3058] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.913321 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3395b9c2d04a6f8a227630087d85d8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:40.913478 776 raft_consensus.cc:2466] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:40.913770 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4b3395b9c2d04a6f8a227630087d85d8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:40.913959 775 raft_consensus.cc:3058] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:40.914111 916 raft_consensus.cc:2466] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:40.914449 988 leader_election.cc:304] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:40.914708 1174 raft_consensus.cc:2802] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:40.914829 1174 raft_consensus.cc:695] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:40.915062 1174 consensus_queue.cc:237] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.919188 455 master_service.cc:948] ReplaceTablet: received request to replace tablet f98bb48081984d23af55d7fa6a8bec7b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.920570 775 raft_consensus.cc:2466] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:40.922353 1029 tablet_service.cc:1468] Processing CreateTablet for tablet dfee69f9484046d8abe10d2f07033e09 (DEFAULT_TABLE table=default.Table_6b615ab3e79749aab1024e49691c3762 [id=40b322522a664b8f8bb8f39388b8ddf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.922473 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a2562570cb0846e19ebc97bdb43bd70d (DEFAULT_TABLE table=default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.922452 892 tablet_service.cc:1468] Processing CreateTablet for tablet dfee69f9484046d8abe10d2f07033e09 (DEFAULT_TABLE table=default.Table_6b615ab3e79749aab1024e49691c3762 [id=40b322522a664b8f8bb8f39388b8ddf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.922835 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfee69f9484046d8abe10d2f07033e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.922890 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfee69f9484046d8abe10d2f07033e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.923036 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2562570cb0846e19ebc97bdb43bd70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.930099 749 tablet_service.cc:1468] Processing CreateTablet for tablet a2562570cb0846e19ebc97bdb43bd70d (DEFAULT_TABLE table=default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.930414 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2562570cb0846e19ebc97bdb43bd70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.932189 892 tablet_service.cc:1468] Processing CreateTablet for tablet a2562570cb0846e19ebc97bdb43bd70d (DEFAULT_TABLE table=default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.932484 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2562570cb0846e19ebc97bdb43bd70d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.934635 1156 ts_tablet_manager.cc:1918] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.934775 1156 log.cc:1199] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/267d9a050fa2414d87a04ba3c91dea03
I20250629 01:51:40.934988 896 tablet_service.cc:1468] Processing CreateTablet for tablet 299922fa5e694bdf96f29f1522068656 (DEFAULT_TABLE table=default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 [id=28e5371a950e42f8803d96b0ffa26cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.935093 1156 ts_tablet_manager.cc:1939] T 267d9a050fa2414d87a04ba3c91dea03 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.935309 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 299922fa5e694bdf96f29f1522068656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.938580 1156 tablet_replica.cc:331] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.938756 1156 raft_consensus.cc:2241] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.938871 1156 raft_consensus.cc:2270] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.941630 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d31db449adde4809af59d2458b8e6118" } new_table_name: "default.Table_661f7318ce384b5bad0d019f704d12c2" modify_external_catalogs: true
I20250629 01:51:40.942785 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_97eb293848a64879af553f06d6660c31" } new_table_name: "default.Table_14b2bea86460403f89586e3ebdbfd084" modify_external_catalogs: true
I20250629 01:51:40.943809 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_447a3977949748f39f7f05ddf762c09d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.944331 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_91eb82c0338648b19d82e9ef4de65851"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.944769 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91eb82c0338648b19d82e9ef4de65851 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.945936 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.946333 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.947479 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_92b93b415dc24417ba11205973730b70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.947526 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ae2d1a922b914ba1bfa2dbca0cda5013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.947877 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92b93b415dc24417ba11205973730b70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:40.949030 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_447a3977949748f39f7f05ddf762c09d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:40.947894 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.950717 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cdb1b42c2c774292b4ee2e323ddee5f6" } modify_external_catalogs: true
I20250629 01:51:40.950852 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cdb1b42c2c774292b4ee2e323ddee5f6" } modify_external_catalogs: true
I20250629 01:51:40.951632 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d68cf4935ca467fa49eb226d95d067a" } modify_external_catalogs: true
I20250629 01:51:40.951763 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d68cf4935ca467fa49eb226d95d067a" } modify_external_catalogs: true
I20250629 01:51:40.952562 1156 ts_tablet_manager.cc:1905] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.953859 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 267d9a050fa2414d87a04ba3c91dea03 (table default.Table_8c79e92b6cfe429088a6b6662f46412c [id=2819c6ba47aa436a9ddd9a72f956c96e]) successfully deleted
W20250629 01:51:40.954111 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6128463cd85d4f0cbfe21b0e5bd0e470 already in progress: creating tablet
W20250629 01:51:40.954303 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1938e67acaf4ab19fcbed13e1f1bfc6 already in progress: creating tablet
W20250629 01:51:40.954502 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d4bd19f9578f4fa19a6daef427e4a6c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4bd19f9578f4fa19a6daef427e4a6c8 already in progress: creating tablet
I20250629 01:51:40.958920 1156 ts_tablet_manager.cc:1918] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.960861 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 299922fa5e694bdf96f29f1522068656 (DEFAULT_TABLE table=default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 [id=28e5371a950e42f8803d96b0ffa26cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.961198 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 299922fa5e694bdf96f29f1522068656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.964388 751 tablet_service.cc:1468] Processing CreateTablet for tablet dfee69f9484046d8abe10d2f07033e09 (DEFAULT_TABLE table=default.Table_6b615ab3e79749aab1024e49691c3762 [id=40b322522a664b8f8bb8f39388b8ddf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.965261 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfee69f9484046d8abe10d2f07033e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.966933 1154 tablet_bootstrap.cc:492] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:40.967067 1154 ts_tablet_manager.cc:1397] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.166s user 0.007s sys 0.004s
I20250629 01:51:40.967767 1154 raft_consensus.cc:357] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.967929 1154 raft_consensus.cc:383] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:40.968021 1154 raft_consensus.cc:738] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:40.971350 1156 log.cc:1199] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4b3395b9c2d04a6f8a227630087d85d8
I20250629 01:51:40.971864 1156 ts_tablet_manager.cc:1939] T 4b3395b9c2d04a6f8a227630087d85d8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:40.973198 753 tablet_service.cc:1468] Processing CreateTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.973421 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e508393b9cb044c4b90194c8aab3184a (DEFAULT_TABLE table=default.Table_b406b018163a43c38fcb2ccfb90c9607 [id=c6883d7c8bd241f78a143cda44abfda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.973578 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a9664bb7a946c48cf79a09955ecad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.973620 889 tablet_service.cc:1468] Processing CreateTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.973743 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e508393b9cb044c4b90194c8aab3184a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.973938 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a9664bb7a946c48cf79a09955ecad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.974651 1156 tablet_replica.cc:331] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:40.974814 1156 raft_consensus.cc:2241] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:40.974970 1156 raft_consensus.cc:2270] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:40.975430 1156 ts_tablet_manager.cc:1905] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:40.977669 754 tablet_service.cc:1468] Processing CreateTablet for tablet e508393b9cb044c4b90194c8aab3184a (DEFAULT_TABLE table=default.Table_b406b018163a43c38fcb2ccfb90c9607 [id=c6883d7c8bd241f78a143cda44abfda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.977667 886 tablet_service.cc:1468] Processing CreateTablet for tablet e508393b9cb044c4b90194c8aab3184a (DEFAULT_TABLE table=default.Table_b406b018163a43c38fcb2ccfb90c9607 [id=c6883d7c8bd241f78a143cda44abfda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.977977 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e508393b9cb044c4b90194c8aab3184a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.977981 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e508393b9cb044c4b90194c8aab3184a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.978001 887 tablet_service.cc:1468] Processing CreateTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 (DEFAULT_TABLE table=default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.978277 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e0de66406e4ce89ce56fa447fe9895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.978885 1158 ts_tablet_manager.cc:1918] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:40.978974 1158 log.cc:1199] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/01eaf627f1dd41c4b5ad6c0a46e2e7cf
I20250629 01:51:40.979281 1158 ts_tablet_manager.cc:1939] T 01eaf627f1dd41c4b5ad6c0a46e2e7cf P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:40.980333 1030 tablet_service.cc:1468] Processing CreateTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 (DEFAULT_TABLE table=default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.980599 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e0de66406e4ce89ce56fa447fe9895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.982172 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4b3395b9c2d04a6f8a227630087d85d8 (table default.Table_0803b366c30642e681bff17044760ace [id=0aeee3c29f1241c4b911b1a390686afb]) successfully deleted
W20250629 01:51:40.982394 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5e9f6de505a34afbae48a373b4b2a0cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e9f6de505a34afbae48a373b4b2a0cf already in progress: creating tablet
W20250629 01:51:40.982587 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 648fd1ca8f514ea3ba4554452b19933f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648fd1ca8f514ea3ba4554452b19933f already in progress: creating tablet
I20250629 01:51:40.982916 749 tablet_service.cc:1468] Processing CreateTablet for tablet 299922fa5e694bdf96f29f1522068656 (DEFAULT_TABLE table=default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 [id=28e5371a950e42f8803d96b0ffa26cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.983237 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 299922fa5e694bdf96f29f1522068656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.984189 750 tablet_service.cc:1468] Processing CreateTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 (DEFAULT_TABLE table=default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.984628 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e0de66406e4ce89ce56fa447fe9895. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:40.985500 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79e61d9576f8414ebf0074d57cdfb1f5 consensus state has no leader
I20250629 01:51:40.986872 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b3395b9c2d04a6f8a227630087d85d8 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:40.987339 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_de085cd6f5f84a5eabb5f1d77be715b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:40.987768 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de085cd6f5f84a5eabb5f1d77be715b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.988516 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ace97daf931b45ac9676c39144d2931f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:40.988862 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:40.988940 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace97daf931b45ac9676c39144d2931f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:40.989168 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2a9664bb7a946c48cf79a09955ecad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.990032 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 5e9f6de505a34afbae48a373b4b2a0cf from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:40.991794 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6688e9665f24e73b80802d47a80a2cd consensus state has no leader
I20250629 01:51:40.994310 1154 consensus_queue.cc:260] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:40.994998 1154 ts_tablet_manager.cc:1428] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20250629 01:51:40.995767 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5f60c48c924d49a186a8c81d154bdc54
I20250629 01:51:40.996837 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 (DEFAULT_TABLE table=default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.997720 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:40.998198 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:40.998431 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:40.998481 893 tablet_service.cc:1468] Processing CreateTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 (DEFAULT_TABLE table=default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.998791 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f60c48c924d49a186a8c81d154bdc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.999040 749 tablet_service.cc:1468] Processing CreateTablet for tablet 36fcb819c61c4fabbb0da295d0002998 (DEFAULT_TABLE table=default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:40.999339 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fcb819c61c4fabbb0da295d0002998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:40.999439 446 catalog_manager.cc:5582] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:40.999825 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6]). Expected version 2 got 0
I20250629 01:51:41.000487 1154 tablet_bootstrap.cc:492] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.001298 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f60c48c924d49a186a8c81d154bdc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.002314 751 tablet_service.cc:1468] Processing CreateTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 (DEFAULT_TABLE table=default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.002452 1154 tablet_bootstrap.cc:654] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.003425 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f60c48c924d49a186a8c81d154bdc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.004387 896 tablet_service.cc:1468] Processing CreateTablet for tablet 36fcb819c61c4fabbb0da295d0002998 (DEFAULT_TABLE table=default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.004868 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fcb819c61c4fabbb0da295d0002998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.007040 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 36fcb819c61c4fabbb0da295d0002998 (DEFAULT_TABLE table=default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.007344 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36fcb819c61c4fabbb0da295d0002998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.009229 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_502b190b8a7046819a8d71733a10c839" } new_table_name: "default.Table_f6f27d5499af49359886879547b0ac31" modify_external_catalogs: true
I20250629 01:51:41.010200 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d7fba939fd154c0ba4dcea2e652fc6aa" } new_table_name: "default.Table_7327f2ea0b414565953b7c1640bc1b59" modify_external_catalogs: true
I20250629 01:51:41.010507 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f (DEFAULT_TABLE table=default.Table_3abcbdb38a38484782a28269e7b69beb [id=29b0e64154664e928993c1035a1bb23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.010780 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba0e267f48a4bf1b58fb2339694719f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.012212 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2fc84dd48d464ce682425fbd41d92b7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.012588 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fc84dd48d464ce682425fbd41d92b7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.013659 888 tablet_service.cc:1468] Processing CreateTablet for tablet 3da0deb865984f70913323364ca38703 (DEFAULT_TABLE table=default.Table_6deb0f6b87e7489d994f1b651f6b5acd [id=80262dd3324a4c2390da9adb2540dd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.013945 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da0deb865984f70913323364ca38703. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.014186 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f (DEFAULT_TABLE table=default.Table_3abcbdb38a38484782a28269e7b69beb [id=29b0e64154664e928993c1035a1bb23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.014271 894 tablet_service.cc:1468] Processing CreateTablet for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (DEFAULT_TABLE table=default.Table_115462b9624b4cdb8e39a8f38af93416 [id=95ccd48727404118a0e2c26dd2103a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.014475 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba0e267f48a4bf1b58fb2339694719f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.014688 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa48f94ac0e48e39578a6e8af5c8e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.020612 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3da0deb865984f70913323364ca38703 (DEFAULT_TABLE table=default.Table_6deb0f6b87e7489d994f1b651f6b5acd [id=80262dd3324a4c2390da9adb2540dd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.020882 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da0deb865984f70913323364ca38703. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.024132 1022 tablet_service.cc:1468] Processing CreateTablet for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (DEFAULT_TABLE table=default.Table_115462b9624b4cdb8e39a8f38af93416 [id=95ccd48727404118a0e2c26dd2103a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.024426 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa48f94ac0e48e39578a6e8af5c8e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.027637 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d46683318dc4c04a617a99f3e620734 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.029435 1157 ts_tablet_manager.cc:1918] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.034363 1158 tablet_replica.cc:331] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.034545 1158 raft_consensus.cc:2241] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:41.034879 1158 pending_rounds.cc:70] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:41.035036 1158 pending_rounds.cc:77] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759146149978112 op_type: NO_OP noop_request { }
I20250629 01:51:41.035159 1158 raft_consensus.cc:2887] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:41.035523 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 01eaf627f1dd41c4b5ad6c0a46e2e7cf (table default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce]) successfully deleted
I20250629 01:51:41.035274 1158 raft_consensus.cc:2270] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.036841 1158 ts_tablet_manager.cc:1905] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.038645 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99a319fbbffa4df69d714b2a48777387" } modify_external_catalogs: true
I20250629 01:51:41.038800 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99a319fbbffa4df69d714b2a48777387" } modify_external_catalogs: true
I20250629 01:51:41.040724 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c0b48fb812c9461092e2d5e0f18b38c6" } modify_external_catalogs: true
I20250629 01:51:41.040867 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c0b48fb812c9461092e2d5e0f18b38c6" } modify_external_catalogs: true
I20250629 01:51:41.034683 1159 tablet_bootstrap.cc:492] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.043867 1159 ts_tablet_manager.cc:1397] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.183s user 0.012s sys 0.015s
I20250629 01:51:41.044551 1159 raft_consensus.cc:357] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.044713 1159 raft_consensus.cc:383] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.044772 1159 raft_consensus.cc:738] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.044965 1159 consensus_queue.cc:260] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.045368 1159 ts_tablet_manager.cc:1428] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:41.045691 1159 tablet_bootstrap.cc:492] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.047645 1159 tablet_bootstrap.cc:654] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.035636 1294 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:41.053275 754 tablet_service.cc:1468] Processing CreateTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f (DEFAULT_TABLE table=default.Table_3abcbdb38a38484782a28269e7b69beb [id=29b0e64154664e928993c1035a1bb23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.058830 1157 log.cc:1199] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0b710052005c4a2aaade1dc60409aef3
I20250629 01:51:41.060364 1157 ts_tablet_manager.cc:1939] T 0b710052005c4a2aaade1dc60409aef3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.062232 1157 tablet_replica.cc:331] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.062866 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0b710052005c4a2aaade1dc60409aef3 (table default.Table_c440da96423948c4bc188695d63152cc [id=64a5b540bb01488c92bb18807a03e2b7]) successfully deleted
I20250629 01:51:41.063334 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cade2a7984a3413a9690af48b8c28e84
I20250629 01:51:41.065354 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.064577 891 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.067023 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba0e267f48a4bf1b58fb2339694719f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.067168 455 master_service.cc:948] ReplaceTablet: received request to replace tablet f1e0de66406e4ce89ce56fa447fe9895 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.068210 1158 ts_tablet_manager.cc:1918] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:41.068369 1158 log.cc:1199] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4b3395b9c2d04a6f8a227630087d85d8
I20250629 01:51:41.068684 891 tablet_service.cc:1515] Processing DeleteTablet for tablet ac26cba8a92945dd9acd10e8a76172ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.068833 1158 ts_tablet_manager.cc:1939] T 4b3395b9c2d04a6f8a227630087d85d8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.068826 752 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.069054 750 tablet_service.cc:1468] Processing CreateTablet for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (DEFAULT_TABLE table=default.Table_115462b9624b4cdb8e39a8f38af93416 [id=95ccd48727404118a0e2c26dd2103a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.069545 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa48f94ac0e48e39578a6e8af5c8e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.066661 1157 raft_consensus.cc:2241] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.069751 1157 raft_consensus.cc:2270] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.068279 753 tablet_service.cc:1468] Processing CreateTablet for tablet 3da0deb865984f70913323364ca38703 (DEFAULT_TABLE table=default.Table_6deb0f6b87e7489d994f1b651f6b5acd [id=80262dd3324a4c2390da9adb2540dd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.070688 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4b3395b9c2d04a6f8a227630087d85d8 (table default.Table_0803b366c30642e681bff17044760ace [id=0aeee3c29f1241c4b911b1a390686afb]) successfully deleted
I20250629 01:51:41.070897 1157 ts_tablet_manager.cc:1905] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.071306 1158 tablet_replica.cc:331] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.071465 1158 raft_consensus.cc:2241] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.071594 1158 raft_consensus.cc:2270] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.073704 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:41.074211 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da0deb865984f70913323364ca38703. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.075912 1295 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:41.076831 1158 ts_tablet_manager.cc:1905] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.078135 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 02d072bf72044dc48d8090660ecb7d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.074424 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 02d072bf72044dc48d8090660ecb7d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.081019 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 02d072bf72044dc48d8090660ecb7d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.082682 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b406b018163a43c38fcb2ccfb90c9607" } new_table_name: "default.Table_92433b26d65e4bf6b433995ad26b2a6b" modify_external_catalogs: true
I20250629 01:51:41.084424 1156 ts_tablet_manager.cc:1918] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.084527 1156 log.cc:1199] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/1185e85345664dfd92c134c5552e7d48
I20250629 01:51:41.085734 1156 ts_tablet_manager.cc:1939] T 1185e85345664dfd92c134c5552e7d48 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.088474 1158 ts_tablet_manager.cc:1918] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.088587 1158 log.cc:1199] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ee70cbe4d80e41b99234f1e1b001434b
I20250629 01:51:41.088881 1158 ts_tablet_manager.cc:1939] T ee70cbe4d80e41b99234f1e1b001434b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.090312 1156 tablet_replica.cc:331] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.091109 1156 raft_consensus.cc:2241] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.091248 1156 raft_consensus.cc:2270] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:41.093884 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bcc3f2a76bc4bae8b306b54e75656cc has no consensus state
I20250629 01:51:41.094775 1227 raft_consensus.cc:491] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.094925 1227 raft_consensus.cc:513] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.095469 1227 leader_election.cc:290] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.096181 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac26cba8a92945dd9acd10e8a76172ca" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:41.096724 1174 raft_consensus.cc:491] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.096861 1051 raft_consensus.cc:1273] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:41.096868 1174 raft_consensus.cc:513] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.097553 1174 leader_election.cc:290] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:41.098089 850 leader_election.cc:343] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.098860 777 raft_consensus.cc:1273] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:41.099231 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20958c5e88454e129aa8efefb63b7f14" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:41.099310 1173 consensus_queue.cc:1035] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:41.099421 916 raft_consensus.cc:2466] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:41.099666 1173 consensus_queue.cc:1035] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:41.101802 1158 tablet_replica.cc:331] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.101992 1158 raft_consensus.cc:2241] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:41.102246 1158 pending_rounds.cc:70] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:41.102208 988 leader_election.cc:304] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:41.102525 1174 raft_consensus.cc:2802] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:41.102598 1174 raft_consensus.cc:491] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:41.102689 1174 raft_consensus.cc:3058] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.102770 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 4b3395b9c2d04a6f8a227630087d85d8 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.104887 1174 raft_consensus.cc:513] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.105417 1174 leader_election.cc:290] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.106570 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20958c5e88454e129aa8efefb63b7f14" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:41.106714 916 raft_consensus.cc:3058] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:41.107698 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f has no consensus state
I20250629 01:51:41.108916 916 raft_consensus.cc:2466] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:41.109041 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac26cba8a92945dd9acd10e8a76172ca" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.109516 988 leader_election.cc:304] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
W20250629 01:51:41.109599 850 leader_election.cc:343] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.109728 850 leader_election.cc:304] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:41.109792 1174 raft_consensus.cc:2802] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:41.109977 1227 raft_consensus.cc:2747] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:41.110270 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20958c5e88454e129aa8efefb63b7f14" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.110428 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ee70cbe4d80e41b99234f1e1b001434b (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:41.110431 776 raft_consensus.cc:2466] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:41.110632 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 1185e85345664dfd92c134c5552e7d48 (table default.Table_07d0ae14810e4c31b91252f2b672ef5d [id=00acceb7e9af4dc8b6fe843e436bf156]) successfully deleted
I20250629 01:51:41.110917 1174 raft_consensus.cc:695] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:41.111215 1174 consensus_queue.cc:237] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.111217 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20958c5e88454e129aa8efefb63b7f14" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:41.111760 774 raft_consensus.cc:3058] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.114409 1159 tablet_bootstrap.cc:492] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.114557 1159 ts_tablet_manager.cc:1397] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.002s
I20250629 01:51:41.114771 774 raft_consensus.cc:2466] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:41.115347 1159 raft_consensus.cc:357] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.115523 1159 raft_consensus.cc:383] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.115617 1159 raft_consensus.cc:738] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.115847 1159 consensus_queue.cc:260] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.118593 1156 ts_tablet_manager.cc:1905] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.119805 1158 pending_rounds.cc:77] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759144993452032 op_type: NO_OP noop_request { }
I20250629 01:51:41.120196 1158 raft_consensus.cc:2887] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:41.120395 1159 ts_tablet_manager.cc:1428] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:41.120590 1158 raft_consensus.cc:2270] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.121636 1158 ts_tablet_manager.cc:1905] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.123637 1159 tablet_bootstrap.cc:492] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.123977 1154 tablet_bootstrap.cc:492] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.124171 1154 ts_tablet_manager.cc:1397] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.004s
I20250629 01:51:41.124859 1154 raft_consensus.cc:357] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.125061 1154 raft_consensus.cc:383] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.125162 1154 raft_consensus.cc:738] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.125308 1159 tablet_bootstrap.cc:654] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.125491 1154 consensus_queue.cc:260] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.126482 1154 ts_tablet_manager.cc:1428] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:41.126609 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f2f4ee457ae9447697c7e33b84157bd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:41.127154 1154 tablet_bootstrap.cc:492] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:41.127140 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2f4ee457ae9447697c7e33b84157bd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.129069 1154 tablet_bootstrap.cc:654] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.129210 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8b976eb77218429b9513d370b45cd5c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.129674 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b976eb77218429b9513d370b45cd5c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.132843 1154 tablet_bootstrap.cc:492] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.134667 447 master_service.cc:948] ReplaceTablet: received request to replace tablet a2a9664bb7a946c48cf79a09955ecad5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.135069 1156 ts_tablet_manager.cc:1918] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.135197 1156 log.cc:1199] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b3de9ab31fc34ba88ac7616dff6e7d5e
I20250629 01:51:41.135464 1156 ts_tablet_manager.cc:1939] T b3de9ab31fc34ba88ac7616dff6e7d5e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.135702 1154 ts_tablet_manager.cc:1397] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250629 01:51:41.136360 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b3de9ab31fc34ba88ac7616dff6e7d5e (table default.Table_8603f3907bdd482ea98c45f0c5cd7903 [id=c95a278758c9465da9582afb968ec8f2]) successfully deleted
I20250629 01:51:41.136955 1154 raft_consensus.cc:357] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.138067 1154 raft_consensus.cc:383] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.138432 1154 raft_consensus.cc:738] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.138715 1154 consensus_queue.cc:260] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.139278 1289 mvcc.cc:204] Tried to move back new op lower bound from 7172759146886344704 to 7172759145294041088. Current Snapshot: MvccSnapshot[applied={T|T < 7172759146886344704}]
I20250629 01:51:41.139803 1154 ts_tablet_manager.cc:1428] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:41.140086 1154 tablet_bootstrap.cc:492] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.142223 1295 tablet.cc:1722] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:41.142591 1294 tablet.cc:1722] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:41.144371 1155 tablet_bootstrap.cc:492] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.144505 1154 tablet_bootstrap.cc:654] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.144508 1155 ts_tablet_manager.cc:1397] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.287s user 0.047s sys 0.043s
I20250629 01:51:41.145246 1155 raft_consensus.cc:357] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.147125 1158 ts_tablet_manager.cc:1918] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:41.149920 1158 log.cc:1199] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/dc6f34d6b31849e190ebfaf1a273d29c
I20250629 01:51:41.150591 1158 ts_tablet_manager.cc:1939] T dc6f34d6b31849e190ebfaf1a273d29c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.150919 1155 raft_consensus.cc:383] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.151021 1155 raft_consensus.cc:738] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.151224 1155 consensus_queue.cc:260] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.152640 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]) successfully deleted
I20250629 01:51:41.153774 1154 tablet_bootstrap.cc:492] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.154073 1156 tablet_replica.cc:331] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.154232 1156 raft_consensus.cc:2241] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.154376 1156 raft_consensus.cc:2270] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.154590 1154 ts_tablet_manager.cc:1397] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250629 01:51:41.154878 1156 ts_tablet_manager.cc:1905] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:41.155414 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5f60c48c924d49a186a8c81d154bdc54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f60c48c924d49a186a8c81d154bdc54 already in progress: creating tablet
I20250629 01:51:41.155725 1158 tablet_replica.cc:331] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.156528 1158 raft_consensus.cc:2241] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.156738 1158 raft_consensus.cc:2270] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.156248 1154 raft_consensus.cc:357] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.157008 1154 raft_consensus.cc:383] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.157622 1154 raft_consensus.cc:738] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.157975 452 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 282fe2a67f864ff4b780a4f3e4e4b40e
I20250629 01:51:41.157876 1154 consensus_queue.cc:260] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.157547 1158 ts_tablet_manager.cc:1905] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.159976 1156 ts_tablet_manager.cc:1918] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.160370 1155 ts_tablet_manager.cc:1428] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250629 01:51:41.161214 1154 ts_tablet_manager.cc:1428] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:51:41.161995 1154 tablet_bootstrap.cc:492] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.162261 1155 tablet_bootstrap.cc:492] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.154088 1298 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:41.163435 1154 tablet_bootstrap.cc:654] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.163846 1155 tablet_bootstrap.cc:654] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:41.164929 440 catalog_manager.cc:5001] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): alter failed for tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]),no further retry: Not found: Tablet not found: dc6f34d6b31849e190ebfaf1a273d29c
I20250629 01:51:41.170790 1155 tablet_bootstrap.cc:492] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.170931 1155 ts_tablet_manager.cc:1397] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20250629 01:51:41.171561 1155 raft_consensus.cc:357] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.171748 1155 raft_consensus.cc:383] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.171821 1155 raft_consensus.cc:738] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.172068 1155 consensus_queue.cc:260] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.173904 1158 ts_tablet_manager.cc:1918] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.174006 1158 log.cc:1199] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0d46683318dc4c04a617a99f3e620734
I20250629 01:51:41.174808 1155 ts_tablet_manager.cc:1428] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:41.174932 1158 ts_tablet_manager.cc:1939] T 0d46683318dc4c04a617a99f3e620734 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.175813 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0d46683318dc4c04a617a99f3e620734 (table default.Table_45d1774983d248aeab1a0353a2d1ece2 [id=f7df5c11e9a84f61b749ebf60a50e6b8]) successfully deleted
I20250629 01:51:41.177259 1155 tablet_bootstrap.cc:492] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:41.177248 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet cade2a7984a3413a9690af48b8c28e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cade2a7984a3413a9690af48b8c28e84 already in progress: creating tablet
I20250629 01:51:41.160075 1156 log.cc:1199] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/dc6f34d6b31849e190ebfaf1a273d29c
W20250629 01:51:41.178362 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 02d072bf72044dc48d8090660ecb7d47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d072bf72044dc48d8090660ecb7d47 already in progress: creating tablet
I20250629 01:51:41.178524 1156 ts_tablet_manager.cc:1939] T dc6f34d6b31849e190ebfaf1a273d29c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.179430 1155 tablet_bootstrap.cc:654] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.181026 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]). Expected version 1 got 0
I20250629 01:51:41.182010 1174 raft_consensus.cc:491] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.182128 1174 raft_consensus.cc:513] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.182636 1174 leader_election.cc:290] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.183248 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac26cba8a92945dd9acd10e8a76172ca" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:41.183449 916 raft_consensus.cc:2466] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:41.184298 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac26cba8a92945dd9acd10e8a76172ca" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.184485 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 034ee15afcef42a9801e39171c0e0e01
W20250629 01:51:41.184725 986 leader_election.cc:343] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.185068 988 leader_election.cc:304] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:41.185340 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]) successfully deleted
I20250629 01:51:41.185606 1174 raft_consensus.cc:2802] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:41.185698 1174 raft_consensus.cc:491] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:41.185765 1174 raft_consensus.cc:3058] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:41.185848 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5f60c48c924d49a186a8c81d154bdc54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f60c48c924d49a186a8c81d154bdc54 already in progress: creating tablet
W20250629 01:51:41.186318 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet cade2a7984a3413a9690af48b8c28e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cade2a7984a3413a9690af48b8c28e84 already in progress: creating tablet
W20250629 01:51:41.186782 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 02d072bf72044dc48d8090660ecb7d47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d072bf72044dc48d8090660ecb7d47 already in progress: creating tablet
I20250629 01:51:41.188380 1174 raft_consensus.cc:513] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.189337 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac26cba8a92945dd9acd10e8a76172ca" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:41.189581 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac26cba8a92945dd9acd10e8a76172ca" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:41.189741 1155 tablet_bootstrap.cc:492] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.189760 916 raft_consensus.cc:3058] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.189836 1155 ts_tablet_manager.cc:1397] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20250629 01:51:41.190361 1155 raft_consensus.cc:357] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.190491 1155 raft_consensus.cc:383] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.190570 1155 raft_consensus.cc:738] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.190743 1155 consensus_queue.cc:260] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.193523 1155 ts_tablet_manager.cc:1428] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20250629 01:51:41.195188 986 leader_election.cc:343] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.195966 1155 tablet_bootstrap.cc:492] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.196677 1159 tablet_bootstrap.cc:492] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.196812 1159 ts_tablet_manager.cc:1397] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20250629 01:51:41.197602 1155 tablet_bootstrap.cc:654] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.198567 916 raft_consensus.cc:2466] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:41.198920 988 leader_election.cc:304] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:41.199184 1272 raft_consensus.cc:2802] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:41.200408 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_175e2bb2b6c5433da19613f5f2e15823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.200906 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_175e2bb2b6c5433da19613f5f2e15823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.201933 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f10d54f76b544b88a632ddf72db73ad7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.202298 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f10d54f76b544b88a632ddf72db73ad7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.203383 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_447a3977949748f39f7f05ddf762c09d" } new_table_name: "default.Table_d20078e8bc4e4a218577b8bc71fc8d44" modify_external_catalogs: true
I20250629 01:51:41.208007 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f1e0de66406e4ce89ce56fa447fe9895
I20250629 01:51:41.208407 455 catalog_manager.cc:6652] ReplaceTablet: tablet f1e0de66406e4ce89ce56fa447fe9895 deleted and replaced by tablet be8e0946a20340b38835c4d8daeaa7b8
I20250629 01:51:41.209134 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 with delete_type TABLET_DATA_DELETED (replaced by tablet be8e0946a20340b38835c4d8daeaa7b8) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.209302 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 with delete_type TABLET_DATA_DELETED (replaced by tablet be8e0946a20340b38835c4d8daeaa7b8) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:41.210204 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f1e0de66406e4ce89ce56fa447fe9895 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e0de66406e4ce89ce56fa447fe9895 already in progress: creating tablet
I20250629 01:51:41.210217 1174 leader_election.cc:290] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.210937 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 with delete_type TABLET_DATA_DELETED (replaced by tablet be8e0946a20340b38835c4d8daeaa7b8) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.213428 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_61b620790cfd40d1a241751cd814dec4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.213968 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b620790cfd40d1a241751cd814dec4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:41.218938 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f1e0de66406e4ce89ce56fa447fe9895 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e0de66406e4ce89ce56fa447fe9895 already in progress: creating tablet
I20250629 01:51:41.219583 1272 raft_consensus.cc:695] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:41.219920 1272 consensus_queue.cc:237] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.220688 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_d1f1005233b74726b856dba70c0a4b8b [id=ac80888442774af495a53bba44dfa763] alter complete (version 3)
I20250629 01:51:41.221671 1154 tablet_bootstrap.cc:492] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.221796 1154 ts_tablet_manager.cc:1397] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20250629 01:51:41.222440 1154 raft_consensus.cc:357] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.222579 1154 raft_consensus.cc:383] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.222651 1154 raft_consensus.cc:738] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.222832 1154 consensus_queue.cc:260] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.223367 1154 ts_tablet_manager.cc:1428] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:41.224506 1154 tablet_bootstrap.cc:492] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.224766 1155 tablet_bootstrap.cc:492] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.224888 1155 ts_tablet_manager.cc:1397] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.001s
I20250629 01:51:41.225488 1155 raft_consensus.cc:357] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.225649 1155 raft_consensus.cc:383] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.225697 1155 raft_consensus.cc:738] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.225885 1155 consensus_queue.cc:260] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.226265 1155 ts_tablet_manager.cc:1428] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:41.226537 1155 tablet_bootstrap.cc:492] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.226872 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6b615ab3e79749aab1024e49691c3762" } new_table_name: "default.Table_2bd4de753ed04021b2e3122b263bafab" modify_external_catalogs: true
I20250629 01:51:41.227262 1154 tablet_bootstrap.cc:654] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.227998 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_30a1da646c914cde85078e3e1270a01e" } modify_external_catalogs: true
I20250629 01:51:41.228200 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_30a1da646c914cde85078e3e1270a01e" } modify_external_catalogs: true
I20250629 01:51:41.230032 1154 tablet_bootstrap.cc:492] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.230178 1154 ts_tablet_manager.cc:1397] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:41.230572 1155 tablet_bootstrap.cc:654] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.230813 1154 raft_consensus.cc:357] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.230962 1154 raft_consensus.cc:383] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.231034 1154 raft_consensus.cc:738] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.232074 1157 ts_tablet_manager.cc:1918] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.232690 1154 consensus_queue.cc:260] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.234510 1308 tablet.cc:1722] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:41.235035 1154 ts_tablet_manager.cc:1428] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:41.236940 1154 tablet_bootstrap.cc:492] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.238418 1154 tablet_bootstrap.cc:654] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.240631 1157 log.cc:1199] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4b3395b9c2d04a6f8a227630087d85d8
I20250629 01:51:41.241477 1157 ts_tablet_manager.cc:1939] T 4b3395b9c2d04a6f8a227630087d85d8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.242324 1157 tablet_replica.cc:331] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.242375 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4b3395b9c2d04a6f8a227630087d85d8 (table default.Table_0803b366c30642e681bff17044760ace [id=0aeee3c29f1241c4b911b1a390686afb]) successfully deleted
I20250629 01:51:41.242483 1157 raft_consensus.cc:2241] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.242640 1157 raft_consensus.cc:2270] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.197901 1159 raft_consensus.cc:357] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.244983 1159 raft_consensus.cc:383] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.245117 1159 raft_consensus.cc:738] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.245409 1159 consensus_queue.cc:260] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.247442 1157 ts_tablet_manager.cc:1905] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.249608 1159 ts_tablet_manager.cc:1428] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20250629 01:51:41.251539 1155 tablet_bootstrap.cc:492] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.251658 1155 ts_tablet_manager.cc:1397] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.010s
I20250629 01:51:41.252332 1155 raft_consensus.cc:357] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.252475 1155 raft_consensus.cc:383] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.252543 1155 raft_consensus.cc:738] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.252732 1155 consensus_queue.cc:260] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.252948 1159 tablet_bootstrap.cc:492] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.255143 1159 tablet_bootstrap.cc:654] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.255328 1155 ts_tablet_manager.cc:1428] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:41.257597 1155 tablet_bootstrap.cc:492] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.258829 1155 tablet_bootstrap.cc:654] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:41.260334 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e508393b9cb044c4b90194c8aab3184a consensus state has no leader
I20250629 01:51:41.265174 1155 tablet_bootstrap.cc:492] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.265305 1155 ts_tablet_manager.cc:1397] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20250629 01:51:41.265816 1155 raft_consensus.cc:357] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.265964 1155 raft_consensus.cc:383] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.266019 1155 raft_consensus.cc:738] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.266204 1155 consensus_queue.cc:260] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.268740 1155 ts_tablet_manager.cc:1428] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:41.271706 1155 tablet_bootstrap.cc:492] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.273315 1155 tablet_bootstrap.cc:654] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.276737 1155 tablet_bootstrap.cc:492] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.276840 1155 ts_tablet_manager.cc:1397] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250629 01:51:41.278273 1155 raft_consensus.cc:357] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.278453 1155 raft_consensus.cc:383] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.278534 1155 raft_consensus.cc:738] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.278736 1155 consensus_queue.cc:260] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.280929 1157 ts_tablet_manager.cc:1918] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.281342 1155 ts_tablet_manager.cc:1428] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:41.283677 1155 tablet_bootstrap.cc:492] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.285123 1155 tablet_bootstrap.cc:654] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.285257 1157 log.cc:1199] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/267d9a050fa2414d87a04ba3c91dea03
I20250629 01:51:41.285648 1157 ts_tablet_manager.cc:1939] T 267d9a050fa2414d87a04ba3c91dea03 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.286662 1157 tablet_replica.cc:331] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.287241 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 267d9a050fa2414d87a04ba3c91dea03 (table default.Table_8c79e92b6cfe429088a6b6662f46412c [id=2819c6ba47aa436a9ddd9a72f956c96e]) successfully deleted
I20250629 01:51:41.287669 1157 raft_consensus.cc:2241] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.287883 1157 raft_consensus.cc:2270] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.289028 1157 ts_tablet_manager.cc:1905] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.293875 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a2a9664bb7a946c48cf79a09955ecad5
I20250629 01:51:41.294309 447 catalog_manager.cc:6652] ReplaceTablet: tablet a2a9664bb7a946c48cf79a09955ecad5 deleted and replaced by tablet 0c712d978a3848b9bb3abdbce8314dec
I20250629 01:51:41.294514 447 master_service.cc:948] ReplaceTablet: received request to replace tablet b3c896a37d9749c5a46bfc7cf7c87911 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.294872 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c712d978a3848b9bb3abdbce8314dec) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.295172 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c712d978a3848b9bb3abdbce8314dec) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:41.296504 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a2a9664bb7a946c48cf79a09955ecad5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2a9664bb7a946c48cf79a09955ecad5 already in progress: creating tablet
W20250629 01:51:41.296741 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a2a9664bb7a946c48cf79a09955ecad5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2a9664bb7a946c48cf79a09955ecad5 already in progress: creating tablet
I20250629 01:51:41.304024 891 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c712d978a3848b9bb3abdbce8314dec) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.314659 451 master_service.cc:948] ReplaceTablet: received request to replace tablet f98bb48081984d23af55d7fa6a8bec7b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.318497 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_399e960d59fe4143843457bd372dbf63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.319137 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_399e960d59fe4143843457bd372dbf63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.326202 1157 ts_tablet_manager.cc:1918] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.326408 1157 log.cc:1199] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/dc6f34d6b31849e190ebfaf1a273d29c
I20250629 01:51:41.326514 1272 raft_consensus.cc:491] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.326687 1272 raft_consensus.cc:513] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.326817 1157 ts_tablet_manager.cc:1939] T dc6f34d6b31849e190ebfaf1a273d29c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.327268 1272 leader_election.cc:290] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.327652 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]) successfully deleted
I20250629 01:51:41.327920 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95c14dacdd0d4156ba7b172edf255ae3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.328085 777 raft_consensus.cc:2466] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
W20250629 01:51:41.328485 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5f60c48c924d49a186a8c81d154bdc54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f60c48c924d49a186a8c81d154bdc54 already in progress: creating tablet
I20250629 01:51:41.328802 986 leader_election.cc:304] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.329134 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95c14dacdd0d4156ba7b172edf255ae3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:41.329608 988 leader_election.cc:343] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:41.329798 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cade2a7984a3413a9690af48b8c28e84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cade2a7984a3413a9690af48b8c28e84 already in progress: creating tablet
I20250629 01:51:41.329838 1157 tablet_replica.cc:331] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.329984 1157 raft_consensus.cc:2241] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.330017 1272 raft_consensus.cc:2802] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:41.330104 1272 raft_consensus.cc:491] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:41.330186 1272 raft_consensus.cc:3058] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.330847 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5e27d74509cc4dc9a4f0566c61348aa3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.331264 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e27d74509cc4dc9a4f0566c61348aa3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.331998 1155 tablet_bootstrap.cc:492] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.332170 1155 ts_tablet_manager.cc:1397] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.049s user 0.023s sys 0.017s
I20250629 01:51:41.332880 1155 raft_consensus.cc:357] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.333035 1155 raft_consensus.cc:383] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.333091 1155 raft_consensus.cc:738] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.330173 1157 raft_consensus.cc:2270] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.333281 1155 consensus_queue.cc:260] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.333671 1155 ts_tablet_manager.cc:1428] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:41.333988 1155 tablet_bootstrap.cc:492] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.333981 1272 raft_consensus.cc:513] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.335958 1155 tablet_bootstrap.cc:654] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.336247 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95c14dacdd0d4156ba7b172edf255ae3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:41.336431 777 raft_consensus.cc:3058] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.336987 1157 ts_tablet_manager.cc:1905] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.338696 1159 tablet_bootstrap.cc:492] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.338806 1159 ts_tablet_manager.cc:1397] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.001s
I20250629 01:51:41.339092 777 raft_consensus.cc:2466] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:41.339501 1159 raft_consensus.cc:357] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.339959 1159 raft_consensus.cc:383] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.340108 1159 raft_consensus.cc:738] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.340215 986 leader_election.cc:304] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.340313 1159 consensus_queue.cc:260] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.340595 1311 raft_consensus.cc:2802] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:41.340708 1311 raft_consensus.cc:695] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:41.341039 1311 consensus_queue.cc:237] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.342108 1159 ts_tablet_manager.cc:1428] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:41.342382 1159 tablet_bootstrap.cc:492] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.346248 1159 tablet_bootstrap.cc:654] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.346685 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95c14dacdd0d4156ba7b172edf255ae3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:41.346937 916 raft_consensus.cc:3058] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.350047 916 raft_consensus.cc:2466] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:41.350346 1263 raft_consensus.cc:491] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.350535 1263 raft_consensus.cc:513] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:41.351119 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a60215417c34bedafbf12172eab469a has no consensus state
I20250629 01:51:41.351182 1263 leader_election.cc:290] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.351864 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6688e9665f24e73b80802d47a80a2cd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:41.352054 1051 raft_consensus.cc:2466] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:41.352787 710 leader_election.cc:304] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.353221 1263 raft_consensus.cc:2802] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:41.353327 1263 raft_consensus.cc:491] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:41.353430 1263 raft_consensus.cc:3058] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.353617 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_adc0a47bc8504a73aa8b9ede02097dd9" } modify_external_catalogs: true
I20250629 01:51:41.353751 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_adc0a47bc8504a73aa8b9ede02097dd9" } modify_external_catalogs: true
I20250629 01:51:41.355151 1157 ts_tablet_manager.cc:1918] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.356725 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6688e9665f24e73b80802d47a80a2cd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:41.357287 915 raft_consensus.cc:2466] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:41.357640 1157 log.cc:1199] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/ac26cba8a92945dd9acd10e8a76172ca
I20250629 01:51:41.357745 1272 leader_election.cc:290] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.358049 1157 ts_tablet_manager.cc:1939] T ac26cba8a92945dd9acd10e8a76172ca P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.358270 1263 raft_consensus.cc:513] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.359012 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6688e9665f24e73b80802d47a80a2cd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:41.359146 915 raft_consensus.cc:3058] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.360680 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_654a8089dcba4f64926b8b405150ffa5" } modify_external_catalogs: true
I20250629 01:51:41.360805 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_654a8089dcba4f64926b8b405150ffa5" } modify_external_catalogs: true
I20250629 01:51:41.361968 915 raft_consensus.cc:2466] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:41.362104 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6688e9665f24e73b80802d47a80a2cd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:41.362257 1051 raft_consensus.cc:3058] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.362689 1154 tablet_bootstrap.cc:492] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.362820 1154 ts_tablet_manager.cc:1397] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.127s user 0.025s sys 0.048s
I20250629 01:51:41.363046 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet ac26cba8a92945dd9acd10e8a76172ca (table default.Table_abbad35fa11f4103ad64ef1ea77dbd65 [id=29e100c61aaf4178b931f15a4467fc63]) successfully deleted
I20250629 01:51:41.363884 1154 raft_consensus.cc:357] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.364023 1154 raft_consensus.cc:383] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.364126 1154 raft_consensus.cc:738] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.364332 1155 tablet_bootstrap.cc:492] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.364290 1154 consensus_queue.cc:260] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.364440 1155 ts_tablet_manager.cc:1397] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.016s
I20250629 01:51:41.364717 1154 ts_tablet_manager.cc:1428] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:41.364801 1051 raft_consensus.cc:2466] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:51:41.364923 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dfee69f9484046d8abe10d2f07033e09 consensus state has no leader
I20250629 01:51:41.365105 1154 tablet_bootstrap.cc:492] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.365053 1155 raft_consensus.cc:357] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.365182 1155 raft_consensus.cc:383] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.365247 1155 raft_consensus.cc:738] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.365414 1155 consensus_queue.cc:260] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.365866 1155 ts_tablet_manager.cc:1428] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:41.366725 710 leader_election.cc:304] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.367020 1164 raft_consensus.cc:2802] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:41.367067 1154 tablet_bootstrap.cc:654] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.367143 1164 raft_consensus.cc:695] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:41.367197 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_877f2c7870d74021bfd9213f7c998e0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:41.367365 1164 consensus_queue.cc:237] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:41.367645 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_877f2c7870d74021bfd9213f7c998e0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.369058 1241 raft_consensus.cc:491] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.369184 1241 raft_consensus.cc:513] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.369793 1241 leader_election.cc:290] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.370844 1155 tablet_bootstrap.cc:492] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.372777 1155 tablet_bootstrap.cc:654] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.374353 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28ea445cec4b35be800dbc8c37af7c" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:41.374774 712 leader_election.cc:343] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.379150 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_bf61fc85ef4b405180f3b4621489590e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.379575 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf61fc85ef4b405180f3b4621489590e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.381337 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28ea445cec4b35be800dbc8c37af7c" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:41.381610 896 tablet_service.cc:1468] Processing CreateTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 (DEFAULT_TABLE table=default.Table_2c04531a50c54791a64d6537b670d54f [id=e4025f53c2ea4b7aa8fafb057c102a3b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:41.381762 710 leader_election.cc:343] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.381884 710 leader_election.cc:304] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:41.381954 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9ab324ce7c488cbaf6e0f0322b5c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.382269 1241 raft_consensus.cc:2747] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:41.382618 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b3c896a37d9749c5a46bfc7cf7c87911
I20250629 01:51:41.383100 447 catalog_manager.cc:6652] ReplaceTablet: tablet b3c896a37d9749c5a46bfc7cf7c87911 deleted and replaced by tablet d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:41.383380 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2ce50be1c3444c059ed91faaf9cd5854" } new_table_name: "default.Table_2ba661f263ce4cbda78e8be06f7d662d" modify_external_catalogs: true
I20250629 01:51:41.383767 1263 leader_election.cc:290] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.385485 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fe6e1e274ae04e02bc6d154720f71759" } new_table_name: "default.Table_5ce4c958f3d4456fb56450ca4cc26221" modify_external_catalogs: true
I20250629 01:51:41.385565 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f (DEFAULT_TABLE table=default.Table_7327f2ea0b414565953b7c1640bc1b59 [id=d5edb2619f57401898104c5d2d036b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.385643 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 575006c12f7c41658e234f7c6c425d69 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.385872 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.386713 891 tablet_service.cc:1468] Processing CreateTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f (DEFAULT_TABLE table=default.Table_7327f2ea0b414565953b7c1640bc1b59 [id=d5edb2619f57401898104c5d2d036b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.386936 894 tablet_service.cc:1468] Processing CreateTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc (DEFAULT_TABLE table=default.Table_f6f27d5499af49359886879547b0ac31 [id=e1d26b8e31044413af1d4bddc58816e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.387317 892 tablet_service.cc:1468] Processing CreateTablet for tablet 955a41750b6a4766bb4fd3f01cfd99c8 (DEFAULT_TABLE table=default.Table_1e32538a1c65431dbe7e57032b44c77e [id=d449d56f1c094e74a6636fca55427749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.387763 886 tablet_service.cc:1468] Processing CreateTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 (DEFAULT_TABLE table=default.Table_737eb1cf8b184a14b253c25d114a14b7 [id=eda52aa556274b26af9896e638b51e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.388221 887 tablet_service.cc:1468] Processing CreateTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 (DEFAULT_TABLE table=default.Table_7768d2d7c9814ba48196e54d12a630a5 [id=82f79023b3c0425c978152033e2183b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.388330 753 tablet_service.cc:1468] Processing CreateTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 (DEFAULT_TABLE table=default.Table_2c04531a50c54791a64d6537b670d54f [id=e4025f53c2ea4b7aa8fafb057c102a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.388653 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9ab324ce7c488cbaf6e0f0322b5c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.388847 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]). Expected version 2 got 0
I20250629 01:51:41.389045 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]). Expected version 1 got 0
I20250629 01:51:41.389894 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]). Expected version 2 got 0
I20250629 01:51:41.390074 446 catalog_manager.cc:5582] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:41.390378 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]). Expected version 2 got 0
I20250629 01:51:41.390918 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358] alter complete (version 1)
I20250629 01:51:41.391511 896 tablet_service.cc:1515] Processing DeleteTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.391454 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_becfa32b77bb40b1b9df61126ffe628e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.391875 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_becfa32b77bb40b1b9df61126ffe628e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.392704 751 tablet_service.cc:1468] Processing CreateTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc (DEFAULT_TABLE table=default.Table_f6f27d5499af49359886879547b0ac31 [id=e1d26b8e31044413af1d4bddc58816e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.387053 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.392992 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bcc3f2a76bc4bae8b306b54e75656cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.393072 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bcc3f2a76bc4bae8b306b54e75656cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.393339 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dfe921a84694903bbdb3fed3d6f2761. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.396205 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.396330 754 tablet_service.cc:1468] Processing CreateTablet for tablet 955a41750b6a4766bb4fd3f01cfd99c8 (DEFAULT_TABLE table=default.Table_1e32538a1c65431dbe7e57032b44c77e [id=d449d56f1c094e74a6636fca55427749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.396631 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955a41750b6a4766bb4fd3f01cfd99c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.399843 1159 tablet_bootstrap.cc:492] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.400007 1159 ts_tablet_manager.cc:1397] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.004s
I20250629 01:51:41.400246 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955a41750b6a4766bb4fd3f01cfd99c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.400460 750 tablet_service.cc:1468] Processing CreateTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 (DEFAULT_TABLE table=default.Table_737eb1cf8b184a14b253c25d114a14b7 [id=eda52aa556274b26af9896e638b51e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.400743 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.400806 1159 raft_consensus.cc:357] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.400979 1159 raft_consensus.cc:383] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.401054 1159 raft_consensus.cc:738] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.401371 1159 consensus_queue.cc:260] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.402715 1159 ts_tablet_manager.cc:1428] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:41.403067 1159 tablet_bootstrap.cc:492] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.404328 752 tablet_service.cc:1468] Processing CreateTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 (DEFAULT_TABLE table=default.Table_7768d2d7c9814ba48196e54d12a630a5 [id=82f79023b3c0425c978152033e2183b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.404592 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dfe921a84694903bbdb3fed3d6f2761. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:41.404830 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 02d072bf72044dc48d8090660ecb7d47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02d072bf72044dc48d8090660ecb7d47 already in progress: creating tablet
W20250629 01:51:41.405494 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f1e0de66406e4ce89ce56fa447fe9895 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e0de66406e4ce89ce56fa447fe9895 already in progress: creating tablet
I20250629 01:51:41.400242 888 tablet_service.cc:1515] Processing DeleteTablet for tablet b3c896a37d9749c5a46bfc7cf7c87911 with delete_type TABLET_DATA_DELETED (replaced by tablet d91fa0f049904a1fb5cbd38d1fd1693a) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:41.409791 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a2a9664bb7a946c48cf79a09955ecad5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2a9664bb7a946c48cf79a09955ecad5 already in progress: creating tablet
I20250629 01:51:41.410140 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f (DEFAULT_TABLE table=default.Table_7327f2ea0b414565953b7c1640bc1b59 [id=d5edb2619f57401898104c5d2d036b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.410355 1157 tablet_replica.cc:331] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.410462 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.410559 1157 raft_consensus.cc:2241] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.410777 1157 raft_consensus.cc:2270] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.411209 1159 tablet_bootstrap.cc:654] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.414953 1027 tablet_service.cc:1468] Processing CreateTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 (DEFAULT_TABLE table=default.Table_2c04531a50c54791a64d6537b670d54f [id=e4025f53c2ea4b7aa8fafb057c102a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.415233 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af9ab324ce7c488cbaf6e0f0322b5c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.417865 894 tablet_service.cc:1468] Processing CreateTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 (DEFAULT_TABLE table=default.Table_91eb82c0338648b19d82e9ef4de65851 [id=700ef08e094b44369154076e65d9ea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.418160 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96324ddf32a1460b8fc37e2c324649e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.418184 890 tablet_service.cc:1468] Processing CreateTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec (DEFAULT_TABLE table=default.Table_fe6e1e274ae04e02bc6d154720f71759 [id=646791dd9bb447fdb2e96a9046dd68d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.418358 889 tablet_service.cc:1468] Processing CreateTablet for tablet 881461aa1bc34dd483f65e171c341bc5 (DEFAULT_TABLE table=default.Table_16db51abc1404d7fa67f1c00a70f9166 [id=a922c0f8e123451fb4060f61472e8d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.418462 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.418589 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc (DEFAULT_TABLE table=default.Table_f6f27d5499af49359886879547b0ac31 [id=e1d26b8e31044413af1d4bddc58816e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.418689 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881461aa1bc34dd483f65e171c341bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.418886 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bcc3f2a76bc4bae8b306b54e75656cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.419824 896 tablet_service.cc:1468] Processing CreateTablet for tablet bedbb041e4094d8399de625badd971bb (DEFAULT_TABLE table=default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.420320 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bedbb041e4094d8399de625badd971bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.422031 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 (DEFAULT_TABLE table=default.Table_7768d2d7c9814ba48196e54d12a630a5 [id=82f79023b3c0425c978152033e2183b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.422323 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5dfe921a84694903bbdb3fed3d6f2761. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.422401 1020 tablet_service.cc:1468] Processing CreateTablet for tablet bedbb041e4094d8399de625badd971bb (DEFAULT_TABLE table=default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.422725 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bedbb041e4094d8399de625badd971bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.422646 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec (DEFAULT_TABLE table=default.Table_fe6e1e274ae04e02bc6d154720f71759 [id=646791dd9bb447fdb2e96a9046dd68d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.423063 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.423684 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad (DEFAULT_TABLE table=default.Table_adc0a47bc8504a73aa8b9ede02097dd9 [id=604220139d4f4b31bfaca78aa2b32f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.424228 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.424369 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 955a41750b6a4766bb4fd3f01cfd99c8 (DEFAULT_TABLE table=default.Table_1e32538a1c65431dbe7e57032b44c77e [id=d449d56f1c094e74a6636fca55427749]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.424674 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 955a41750b6a4766bb4fd3f01cfd99c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.425176 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0b7a640451934bff8672cd88dc86c529 (DEFAULT_TABLE table=default.Table_ed3568508cfb4c4b9297f986d7a9963e [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.425458 893 tablet_service.cc:1468] Processing CreateTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 (DEFAULT_TABLE table=default.Table_92b93b415dc24417ba11205973730b70 [id=a07cbcbb14984c538bdd7d1441d52055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.426245 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7a640451934bff8672cd88dc86c529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.428332 1031 tablet_service.cc:1468] Processing CreateTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 (DEFAULT_TABLE table=default.Table_737eb1cf8b184a14b253c25d114a14b7 [id=eda52aa556274b26af9896e638b51e7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.428609 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.429080 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c7f2ac44734a39ab6434a74c91b9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.432212 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b3c896a37d9749c5a46bfc7cf7c87911 with delete_type TABLET_DATA_DELETED (replaced by tablet d91fa0f049904a1fb5cbd38d1fd1693a) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.433455 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d61687e19d994e87b09e55bd3b9476d7 (DEFAULT_TABLE table=default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb [id=f274d40e73944253825a87b39ee56b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.433710 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61687e19d994e87b09e55bd3b9476d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.433923 1157 ts_tablet_manager.cc:1905] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.436373 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 (DEFAULT_TABLE table=default.Table_91eb82c0338648b19d82e9ef4de65851 [id=700ef08e094b44369154076e65d9ea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.436741 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96324ddf32a1460b8fc37e2c324649e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.440438 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 881461aa1bc34dd483f65e171c341bc5 (DEFAULT_TABLE table=default.Table_16db51abc1404d7fa67f1c00a70f9166 [id=a922c0f8e123451fb4060f61472e8d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.440824 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881461aa1bc34dd483f65e171c341bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.445820 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 (DEFAULT_TABLE table=default.Table_92b93b415dc24417ba11205973730b70 [id=a07cbcbb14984c538bdd7d1441d52055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.446185 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c7f2ac44734a39ab6434a74c91b9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.448508 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 0b7a640451934bff8672cd88dc86c529 (DEFAULT_TABLE table=default.Table_ed3568508cfb4c4b9297f986d7a9963e [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.449018 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7a640451934bff8672cd88dc86c529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.449560 1158 tablet_replica.cc:331] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.449810 1158 raft_consensus.cc:2241] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.450086 1158 raft_consensus.cc:2270] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.418375 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad (DEFAULT_TABLE table=default.Table_adc0a47bc8504a73aa8b9ede02097dd9 [id=604220139d4f4b31bfaca78aa2b32f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.458339 1158 ts_tablet_manager.cc:1905] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.471066 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.489355 888 tablet_service.cc:1468] Processing CreateTablet for tablet d61687e19d994e87b09e55bd3b9476d7 (DEFAULT_TABLE table=default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb [id=f274d40e73944253825a87b39ee56b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.494699 1157 ts_tablet_manager.cc:1918] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.501929 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61687e19d994e87b09e55bd3b9476d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.502599 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b4701123dfa467ebb6c95f9742b3874) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.502871 1157 log.cc:1199] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b3de9ab31fc34ba88ac7616dff6e7d5e
I20250629 01:51:41.502902 1156 tablet_replica.cc:331] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.504966 757 tablet_service.cc:1515] Processing DeleteTablet for tablet b3c896a37d9749c5a46bfc7cf7c87911 with delete_type TABLET_DATA_DELETED (replaced by tablet d91fa0f049904a1fb5cbd38d1fd1693a) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.505154 1156 raft_consensus.cc:2241] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.505314 1156 raft_consensus.cc:2270] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.507104 1156 ts_tablet_manager.cc:1905] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.512517 1157 ts_tablet_manager.cc:1939] T b3de9ab31fc34ba88ac7616dff6e7d5e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.512953 1157 tablet_replica.cc:331] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.513028 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b3de9ab31fc34ba88ac7616dff6e7d5e (table default.Table_8603f3907bdd482ea98c45f0c5cd7903 [id=c95a278758c9465da9582afb968ec8f2]) successfully deleted
I20250629 01:51:41.513139 1157 raft_consensus.cc:2241] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:41.513518 1157 raft_consensus.cc:2270] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.514249 1157 ts_tablet_manager.cc:1905] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.515782 757 tablet_service.cc:1468] Processing CreateTablet for tablet d61687e19d994e87b09e55bd3b9476d7 (DEFAULT_TABLE table=default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb [id=f274d40e73944253825a87b39ee56b71]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:41.517410 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20250629 01:51:41.520476 1157 ts_tablet_manager.cc:1918] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:41.526433 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 36fcb819c61c4fabbb0da295d0002998
I20250629 01:51:41.526973 1158 ts_tablet_manager.cc:1918] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:41.527093 1158 log.cc:1199] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b3c896a37d9749c5a46bfc7cf7c87911
I20250629 01:51:41.527159 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9edf5b31c7b24325930accee8cb627c3" } modify_external_catalogs: true
I20250629 01:51:41.527335 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9edf5b31c7b24325930accee8cb627c3" } modify_external_catalogs: true
I20250629 01:51:41.527479 1158 ts_tablet_manager.cc:1939] T b3c896a37d9749c5a46bfc7cf7c87911 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.528497 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad
I20250629 01:51:41.528996 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61687e19d994e87b09e55bd3b9476d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.529124 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_16db51abc1404d7fa67f1c00a70f9166" } modify_external_catalogs: true
I20250629 01:51:41.529160 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92b93b415dc24417ba11205973730b70" } new_table_name: "default.Table_fa24ba8423a34245b118894025a8b4f4" modify_external_catalogs: true
I20250629 01:51:41.529295 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_16db51abc1404d7fa67f1c00a70f9166" } modify_external_catalogs: true
I20250629 01:51:41.530144 756 tablet_service.cc:1468] Processing CreateTablet for tablet bedbb041e4094d8399de625badd971bb (DEFAULT_TABLE table=default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.530361 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 582e063f97ab415cab126845789e6e20
I20250629 01:51:41.530465 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bedbb041e4094d8399de625badd971bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.530962 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0735cd5003d948d68959b946189deca4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:41.531097 755 tablet_service.cc:1468] Processing CreateTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 (DEFAULT_TABLE table=default.Table_91eb82c0338648b19d82e9ef4de65851 [id=700ef08e094b44369154076e65d9ea94]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:41.531378 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0735cd5003d948d68959b946189deca4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:41.532402 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe14d1d54db74a99b05cf8b1a60827a9 consensus state has no leader
I20250629 01:51:41.532238 748 tablet_service.cc:1468] Processing CreateTablet for tablet 881461aa1bc34dd483f65e171c341bc5 (DEFAULT_TABLE table=default.Table_16db51abc1404d7fa67f1c00a70f9166 [id=a922c0f8e123451fb4060f61472e8d41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.532713 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_81bc81a3198343948ffbd72f6d683481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.533108 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81bc81a3198343948ffbd72f6d683481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.533802 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96324ddf32a1460b8fc37e2c324649e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:41.534044 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75fd4ffcf896419b81a1c0eb7c5aeeec consensus state has no leader
I20250629 01:51:41.534221 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 582e063f97ab415cab126845789e6e20 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.534667 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a726ca476374467fb904639d799b7a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.535066 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a726ca476374467fb904639d799b7a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.536150 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 575006c12f7c41658e234f7c6c425d69
I20250629 01:51:41.536235 747 tablet_service.cc:1468] Processing CreateTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 (DEFAULT_TABLE table=default.Table_92b93b415dc24417ba11205973730b70 [id=a07cbcbb14984c538bdd7d1441d52055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.536600 450 catalog_manager.cc:6652] ReplaceTablet: tablet 575006c12f7c41658e234f7c6c425d69 deleted and replaced by tablet a792a26725954adea15ad57e31bfb224
I20250629 01:51:41.536654 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c7f2ac44734a39ab6434a74c91b9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.536769 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 05905638c6e241a095438f277f53adab from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.537251 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c6543e800e6f41d4ac45569c0650a8f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.537670 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6543e800e6f41d4ac45569c0650a8f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.541283 1157 log.cc:1199] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b3c896a37d9749c5a46bfc7cf7c87911
I20250629 01:51:41.541630 1157 ts_tablet_manager.cc:1939] T b3c896a37d9749c5a46bfc7cf7c87911 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.543689 1159 tablet_bootstrap.cc:492] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.543804 1159 ts_tablet_manager.cc:1397] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.141s user 0.024s sys 0.024s
I20250629 01:51:41.544736 746 tablet_service.cc:1468] Processing CreateTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec (DEFAULT_TABLE table=default.Table_fe6e1e274ae04e02bc6d154720f71759 [id=646791dd9bb447fdb2e96a9046dd68d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.544991 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]). Expected version 2 got 0
I20250629 01:51:41.546759 1159 raft_consensus.cc:357] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.546901 1159 raft_consensus.cc:383] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.546984 1159 raft_consensus.cc:738] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.547171 1159 consensus_queue.cc:260] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.547538 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 79e61d9576f8414ebf0074d57cdfb1f5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.547884 1159 ts_tablet_manager.cc:1428] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:41.548132 1159 tablet_bootstrap.cc:492] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.549638 1159 tablet_bootstrap.cc:654] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.549867 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]) successfully deleted
I20250629 01:51:41.550789 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.550993 1157 tablet_replica.cc:331] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.551164 1157 raft_consensus.cc:2241] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.551295 1157 raft_consensus.cc:2270] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.551903 1157 ts_tablet_manager.cc:1905] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.552309 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.556483 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 582e063f97ab415cab126845789e6e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.562592 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 881461aa1bc34dd483f65e171c341bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.563922 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.564116 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6f7d1b03023842daa7f9d43483dc6b95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.564649 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f7d1b03023842daa7f9d43483dc6b95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.564719 1154 tablet_bootstrap.cc:492] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.564848 1154 ts_tablet_manager.cc:1397] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.200s user 0.028s sys 0.049s
I20250629 01:51:41.565466 1154 raft_consensus.cc:357] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.565631 1154 raft_consensus.cc:383] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.567188 1154 raft_consensus.cc:738] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.567392 1154 consensus_queue.cc:260] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.568198 895 tablet_service.cc:1515] Processing DeleteTablet for tablet d550a0ce24e24055947c8344c79bcca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.568447 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b4701123dfa467ebb6c95f9742b3874) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.568598 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 36fcb819c61c4fabbb0da295d0002998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.568733 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 575006c12f7c41658e234f7c6c425d69 with delete_type TABLET_DATA_DELETED (replaced by tablet a792a26725954adea15ad57e31bfb224) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.569357 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_61b620790cfd40d1a241751cd814dec4" } new_table_name: "default.Table_ca28c370cb5e435b9dc8db3efb233a44" modify_external_catalogs: true
W20250629 01:51:41.569613 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 52164us
I20250629 01:51:41.570806 1157 ts_tablet_manager.cc:1918] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.571733 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.571971 1154 ts_tablet_manager.cc:1428] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:41.572541 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 582e063f97ab415cab126845789e6e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.572703 1154 tablet_bootstrap.cc:492] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.573850 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 575006c12f7c41658e234f7c6c425d69 with delete_type TABLET_DATA_DELETED (replaced by tablet a792a26725954adea15ad57e31bfb224) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.574553 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a22af67c80541b38f2d4c121fdeb6e4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5b4701123dfa467ebb6c95f9742b3874) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.574714 1154 tablet_bootstrap.cc:654] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.575466 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet ac26cba8a92945dd9acd10e8a76172ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.571513 1159 tablet_bootstrap.cc:492] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.575959 1159 ts_tablet_manager.cc:1397] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.006s
I20250629 01:51:41.576431 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.576546 1159 raft_consensus.cc:357] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.576721 1159 raft_consensus.cc:383] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.576799 1159 raft_consensus.cc:738] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.576896 1311 raft_consensus.cc:491] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.577023 1311 raft_consensus.cc:513] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.577008 1159 consensus_queue.cc:260] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.577523 1311 leader_election.cc:290] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.577546 1159 ts_tablet_manager.cc:1428] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:41.577840 1159 tablet_bootstrap.cc:492] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.579820 1159 tablet_bootstrap.cc:654] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.580667 1157 log.cc:1199] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/95c14dacdd0d4156ba7b172edf255ae3
I20250629 01:51:41.581020 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "431dc961cdcf4cd38b521d8d8fc6f406" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:41.581338 1157 ts_tablet_manager.cc:1939] T 95c14dacdd0d4156ba7b172edf255ae3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.582316 1157 tablet_replica.cc:331] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.582477 1157 raft_consensus.cc:2241] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.582633 1157 raft_consensus.cc:2270] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.583148 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 95c14dacdd0d4156ba7b172edf255ae3 (table default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]) successfully deleted
W20250629 01:51:41.583388 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad already in progress: creating tablet
W20250629 01:51:41.583634 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 582e063f97ab415cab126845789e6e20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 582e063f97ab415cab126845789e6e20 already in progress: creating tablet
I20250629 01:51:41.577225 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.584120 1158 tablet_replica.cc:331] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:41.584203 988 leader_election.cc:343] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.584334 1158 raft_consensus.cc:2241] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.584507 1158 raft_consensus.cc:2270] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.585258 1158 ts_tablet_manager.cc:1905] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.585304 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 582e063f97ab415cab126845789e6e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.586167 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 575006c12f7c41658e234f7c6c425d69 with delete_type TABLET_DATA_DELETED (replaced by tablet a792a26725954adea15ad57e31bfb224) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.587605 1157 ts_tablet_manager.cc:1905] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.588209 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.588521 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "431dc961cdcf4cd38b521d8d8fc6f406" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:41.589476 986 leader_election.cc:343] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.589567 986 leader_election.cc:304] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:41.589880 1311 raft_consensus.cc:2747] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:41.589857 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]) successfully deleted
I20250629 01:51:41.590133 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 36fcb819c61c4fabbb0da295d0002998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:41.590641 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad already in progress: creating tablet
W20250629 01:51:41.590842 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 582e063f97ab415cab126845789e6e20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 582e063f97ab415cab126845789e6e20 already in progress: creating tablet
W20250629 01:51:41.591027 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 575006c12f7c41658e234f7c6c425d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575006c12f7c41658e234f7c6c425d69 already in progress: creating tablet
I20250629 01:51:41.591346 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.602727 1159 tablet_bootstrap.cc:492] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.602871 1159 ts_tablet_manager.cc:1397] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.000s
I20250629 01:51:41.603690 1159 raft_consensus.cc:357] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.603879 1159 raft_consensus.cc:383] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.603962 1159 raft_consensus.cc:738] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.604780 1156 ts_tablet_manager.cc:1918] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.605057 1159 consensus_queue.cc:260] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.605964 1154 tablet_bootstrap.cc:492] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.606065 1154 ts_tablet_manager.cc:1397] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.005s
I20250629 01:51:41.606617 1156 log.cc:1199] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4a22af67c80541b38f2d4c121fdeb6e4
I20250629 01:51:41.606930 1154 raft_consensus.cc:357] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.607028 1156 ts_tablet_manager.cc:1939] T 4a22af67c80541b38f2d4c121fdeb6e4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.607097 1154 raft_consensus.cc:383] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.607182 1154 raft_consensus.cc:738] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.607383 1154 consensus_queue.cc:260] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.607460 1158 ts_tablet_manager.cc:1918] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.607620 1158 log.cc:1199] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4a22af67c80541b38f2d4c121fdeb6e4
I20250629 01:51:41.608004 1154 ts_tablet_manager.cc:1428] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:41.608146 1158 ts_tablet_manager.cc:1939] T 4a22af67c80541b38f2d4c121fdeb6e4 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.608152 1159 ts_tablet_manager.cc:1428] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250629 01:51:41.609467 1159 tablet_bootstrap.cc:492] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.609505 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 36fcb819c61c4fabbb0da295d0002998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.610167 1157 ts_tablet_manager.cc:1918] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.610327 1157 log.cc:1199] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d550a0ce24e24055947c8344c79bcca3
I20250629 01:51:41.610759 1157 ts_tablet_manager.cc:1939] T d550a0ce24e24055947c8344c79bcca3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.611441 1159 tablet_bootstrap.cc:654] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.612195 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d550a0ce24e24055947c8344c79bcca3 (table default.Table_daa3abe221c44551848996265764f670 [id=5a1fd538766d4b189cb5dd04bf0b2314]) successfully deleted
I20250629 01:51:41.612607 1157 tablet_replica.cc:331] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.612957 1157 raft_consensus.cc:2241] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.613283 1157 raft_consensus.cc:2270] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.614051 1157 ts_tablet_manager.cc:1905] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.615283 1156 tablet_replica.cc:331] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.615526 1156 raft_consensus.cc:2241] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.615900 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4a22af67c80541b38f2d4c121fdeb6e4 (table default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
I20250629 01:51:41.616473 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4a22af67c80541b38f2d4c121fdeb6e4 (table default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
I20250629 01:51:41.616684 1156 raft_consensus.cc:2270] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.559473 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0b7a640451934bff8672cd88dc86c529 (DEFAULT_TABLE table=default.Table_ed3568508cfb4c4b9297f986d7a9963e [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.617517 1156 ts_tablet_manager.cc:1905] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.617798 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b7a640451934bff8672cd88dc86c529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.560528 744 tablet_service.cc:1468] Processing CreateTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad (DEFAULT_TABLE table=default.Table_adc0a47bc8504a73aa8b9ede02097dd9 [id=604220139d4f4b31bfaca78aa2b32f29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.620788 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.622922 1157 ts_tablet_manager.cc:1918] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.623021 1157 log.cc:1199] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4a22af67c80541b38f2d4c121fdeb6e4
I20250629 01:51:41.623276 1157 ts_tablet_manager.cc:1939] T 4a22af67c80541b38f2d4c121fdeb6e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.625089 1159 tablet_bootstrap.cc:492] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.625240 1159 ts_tablet_manager.cc:1397] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20250629 01:51:41.626020 1159 raft_consensus.cc:357] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.626221 1159 raft_consensus.cc:383] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.626287 1159 raft_consensus.cc:738] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.627234 1051 raft_consensus.cc:1273] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:41.627424 915 raft_consensus.cc:1273] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:41.626614 1159 consensus_queue.cc:260] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.627841 1241 consensus_queue.cc:1035] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:41.628432 1159 ts_tablet_manager.cc:1428] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:41.628698 1159 tablet_bootstrap.cc:492] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.630646 1263 consensus_queue.cc:1035] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:41.630877 1159 tablet_bootstrap.cc:654] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.632731 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4a22af67c80541b38f2d4c121fdeb6e4 (table default.Table_04985f57403a42598f1c7ef473177a7b [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
W20250629 01:51:41.632931 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 36fcb819c61c4fabbb0da295d0002998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36fcb819c61c4fabbb0da295d0002998 already in progress: creating tablet
W20250629 01:51:41.633122 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 575006c12f7c41658e234f7c6c425d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575006c12f7c41658e234f7c6c425d69 already in progress: creating tablet
I20250629 01:51:41.648159 1155 tablet_bootstrap.cc:492] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.648353 1155 ts_tablet_manager.cc:1397] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.278s user 0.027s sys 0.027s
I20250629 01:51:41.649192 1155 raft_consensus.cc:357] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.649446 1155 raft_consensus.cc:383] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.649528 1155 raft_consensus.cc:738] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.649775 1155 consensus_queue.cc:260] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.650354 1155 ts_tablet_manager.cc:1428] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:41.650672 1155 tablet_bootstrap.cc:492] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.652724 1155 tablet_bootstrap.cc:654] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.653371 1159 tablet_bootstrap.cc:492] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.653527 1159 ts_tablet_manager.cc:1397] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.008s
I20250629 01:51:41.654191 1159 raft_consensus.cc:357] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.654343 1159 raft_consensus.cc:383] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.654428 1159 raft_consensus.cc:738] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.655730 1159 consensus_queue.cc:260] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.656347 1338 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:41.609834 1158 tablet_replica.cc:331] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.609980 1154 tablet_bootstrap.cc:492] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.658708 1158 raft_consensus.cc:2241] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:41.658901 1158 pending_rounds.cc:70] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:41.658972 1158 pending_rounds.cc:77] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759147400400896 op_type: NO_OP noop_request { }
I20250629 01:51:41.659054 1158 raft_consensus.cc:2887] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:41.659153 1158 raft_consensus.cc:2270] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.660043 1154 tablet_bootstrap.cc:654] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.660979 1159 ts_tablet_manager.cc:1428] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:41.661207 1159 tablet_bootstrap.cc:492] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:41.661345 1158 ts_tablet_manager.cc:1905] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.666182 1156 ts_tablet_manager.cc:1918] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:41.666240 1159 tablet_bootstrap.cc:654] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.666288 1156 log.cc:1199] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b3c896a37d9749c5a46bfc7cf7c87911
I20250629 01:51:41.666688 1156 ts_tablet_manager.cc:1939] T b3c896a37d9749c5a46bfc7cf7c87911 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.667474 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b3c896a37d9749c5a46bfc7cf7c87911 (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]) successfully deleted
I20250629 01:51:41.668434 1339 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:41.671947 1158 ts_tablet_manager.cc:1918] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250629 01:51:41.672475 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7c7f2ac44734a39ab6434a74c91b9e4 consensus state has no leader
I20250629 01:51:41.674453 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_bba7ed21c41c46ca8f68ce01c5225b1e [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 2 got 0
W20250629 01:51:41.676255 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf4b52fae47f45ad9d77b0a97a854860 has no consensus state
I20250629 01:51:41.676589 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_877f2c7870d74021bfd9213f7c998e0d" } new_table_name: "default.Table_b5f98a84fe7341ac8e6b11b04066323c" modify_external_catalogs: true
I20250629 01:51:41.676930 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7748846a3df14ffa801321ccde86e35f
I20250629 01:51:41.677340 449 catalog_manager.cc:5582] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:41.677560 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f0a26d9ea3e8407da5ca0813db26beb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:41.677739 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]). Expected version 2 got 0
I20250629 01:51:41.672078 1158 log.cc:1199] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ac26cba8a92945dd9acd10e8a76172ca
W20250629 01:51:41.678033 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0a26d9ea3e8407da5ca0813db26beb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.678380 1158 ts_tablet_manager.cc:1939] T ac26cba8a92945dd9acd10e8a76172ca P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.679136 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 881461aa1bc34dd483f65e171c341bc5
I20250629 01:51:41.679231 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.679507 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 7748846a3df14ffa801321ccde86e35f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.679961 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 7748846a3df14ffa801321ccde86e35f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.679921 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3906fb981a8640e1906a78a321c479b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.680485 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3906fb981a8640e1906a78a321c479b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.682056 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 96324ddf32a1460b8fc37e2c324649e6 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.683480 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_58446e63991843cea406fca1f041d6ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.683995 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58446e63991843cea406fca1f041d6ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.685407 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 79e61d9576f8414ebf0074d57cdfb1f5
I20250629 01:51:41.685885 448 catalog_manager.cc:6652] ReplaceTablet: tablet 79e61d9576f8414ebf0074d57cdfb1f5 deleted and replaced by tablet e82b1352bb4d49769808dc402c85890f
I20250629 01:51:41.687134 1154 tablet_bootstrap.cc:492] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.687279 1154 ts_tablet_manager.cc:1397] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.000s
I20250629 01:51:41.687929 1154 raft_consensus.cc:357] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.688072 1154 raft_consensus.cc:383] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.688154 1154 raft_consensus.cc:738] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.688333 1158 tablet_replica.cc:331] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.688385 1154 consensus_queue.cc:260] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.688509 1158 raft_consensus.cc:2241] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:41.688688 1158 pending_rounds.cc:70] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:41.688753 1158 pending_rounds.cc:77] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759147895799808 op_type: NO_OP noop_request { }
I20250629 01:51:41.688850 1158 raft_consensus.cc:2887] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:41.688951 1158 raft_consensus.cc:2270] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.690168 1154 ts_tablet_manager.cc:1428] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:41.690721 1158 ts_tablet_manager.cc:1905] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.691164 1154 tablet_bootstrap.cc:492] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.692368 1263 raft_consensus.cc:491] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.692497 1263 raft_consensus.cc:513] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.693048 1263 leader_election.cc:290] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.693095 1154 tablet_bootstrap.cc:654] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:41.694080 712 leader_election.cc:343] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: ac26cba8a92945dd9acd10e8a76172ca
W20250629 01:51:41.694268 710 leader_election.cc:343] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ac26cba8a92945dd9acd10e8a76172ca
I20250629 01:51:41.694366 710 leader_election.cc:304] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:41.694661 1263 raft_consensus.cc:2747] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:41.696709 1158 ts_tablet_manager.cc:1918] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:41.696810 1158 log.cc:1199] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/95c14dacdd0d4156ba7b172edf255ae3
I20250629 01:51:41.697480 1158 ts_tablet_manager.cc:1939] T 95c14dacdd0d4156ba7b172edf255ae3 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.698173 1158 tablet_replica.cc:331] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.698320 1158 raft_consensus.cc:2241] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.698444 1158 raft_consensus.cc:2270] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:41.698621 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd709b557b334434a6c2c4a60a996342 has no consensus state
I20250629 01:51:41.705283 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6]). Expected version 2 got 0
I20250629 01:51:41.705390 1158 ts_tablet_manager.cc:1905] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.710475 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 96324ddf32a1460b8fc37e2c324649e6
I20250629 01:51:41.710795 1308 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:41.710927 454 catalog_manager.cc:6652] ReplaceTablet: tablet 96324ddf32a1460b8fc37e2c324649e6 deleted and replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de
W20250629 01:51:41.679947 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 582e063f97ab415cab126845789e6e20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 582e063f97ab415cab126845789e6e20 already in progress: creating tablet
W20250629 01:51:41.720311 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 575006c12f7c41658e234f7c6c425d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 575006c12f7c41658e234f7c6c425d69 already in progress: creating tablet
W20250629 01:51:41.720754 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad already in progress: creating tablet
W20250629 01:51:41.721031 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 36fcb819c61c4fabbb0da295d0002998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36fcb819c61c4fabbb0da295d0002998 already in progress: creating tablet
W20250629 01:51:41.721280 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7748846a3df14ffa801321ccde86e35f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7748846a3df14ffa801321ccde86e35f already in progress: creating tablet
I20250629 01:51:41.721460 1158 ts_tablet_manager.cc:1918] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.721572 1158 log.cc:1199] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/431dc961cdcf4cd38b521d8d8fc6f406
I20250629 01:51:41.721863 1158 ts_tablet_manager.cc:1939] T 431dc961cdcf4cd38b521d8d8fc6f406 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.722095 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 881461aa1bc34dd483f65e171c341bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.723165 1154 tablet_bootstrap.cc:492] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.723273 1154 ts_tablet_manager.cc:1397] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250629 01:51:41.723399 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 with delete_type TABLET_DATA_DELETED (replaced by tablet e82b1352bb4d49769808dc402c85890f) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.723853 1154 raft_consensus.cc:357] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.724000 1154 raft_consensus.cc:383] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.724087 1154 raft_consensus.cc:738] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.724274 1154 consensus_queue.cc:260] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.724465 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.724684 1154 ts_tablet_manager.cc:1428] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:41.724704 1156 tablet_replica.cc:331] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.724923 1154 tablet_bootstrap.cc:492] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.724936 1156 raft_consensus.cc:2241] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.725093 1156 raft_consensus.cc:2270] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.725749 746 tablet_service.cc:1515] Processing DeleteTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.726883 1154 tablet_bootstrap.cc:654] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.728395 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.728937 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 881461aa1bc34dd483f65e171c341bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.729035 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d550a0ce24e24055947c8344c79bcca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.729112 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 with delete_type TABLET_DATA_DELETED (replaced by tablet e82b1352bb4d49769808dc402c85890f) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.728937 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d550a0ce24e24055947c8344c79bcca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.729593 746 tablet_service.cc:1515] Processing DeleteTablet for tablet ac26cba8a92945dd9acd10e8a76172ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.730094 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 95c14dacdd0d4156ba7b172edf255ae3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.730566 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.731797 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.732069 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27" } modify_external_catalogs: true
I20250629 01:51:41.732281 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27" } modify_external_catalogs: true
I20250629 01:51:41.732824 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1fb69ddfd0b2426abea3a0352dddb3a7" } modify_external_catalogs: true
I20250629 01:51:41.732969 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1fb69ddfd0b2426abea3a0352dddb3a7" } modify_external_catalogs: true
I20250629 01:51:41.733296 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ac26cba8a92945dd9acd10e8a76172ca (table default.Table_abbad35fa11f4103ad64ef1ea77dbd65 [id=29e100c61aaf4178b931f15a4467fc63]) successfully deleted
I20250629 01:51:41.733506 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 95c14dacdd0d4156ba7b172edf255ae3 (table default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]) successfully deleted
I20250629 01:51:41.733660 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 431dc961cdcf4cd38b521d8d8fc6f406 (table default.Table_a37f4562238b452abf51e6e3d547ca36 [id=c6a48bb9a6874c40b2f7679f2db9e756]) successfully deleted
I20250629 01:51:41.733711 1156 ts_tablet_manager.cc:1905] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:41.733871 440 catalog_manager.cc:5001] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): alter failed for tablet dc6f34d6b31849e190ebfaf1a273d29c (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]),no further retry: Not found: Tablet not found: dc6f34d6b31849e190ebfaf1a273d29c
W20250629 01:51:41.734099 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 881461aa1bc34dd483f65e171c341bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 881461aa1bc34dd483f65e171c341bc5 already in progress: creating tablet
W20250629 01:51:41.734287 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 79e61d9576f8414ebf0074d57cdfb1f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e61d9576f8414ebf0074d57cdfb1f5 already in progress: creating tablet
I20250629 01:51:41.734547 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ace97daf931b45ac9676c39144d2931f" } modify_external_catalogs: true
I20250629 01:51:41.734675 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ace97daf931b45ac9676c39144d2931f" } modify_external_catalogs: true
I20250629 01:51:41.740293 1311 raft_consensus.cc:491] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.740433 1311 raft_consensus.cc:513] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.740934 1311 leader_election.cc:290] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:41.741611 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4bd19f9578f4fa19a6daef427e4a6c8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:41.742012 986 leader_election.cc:343] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:41.742250 1156 ts_tablet_manager.cc:1918] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.742405 1156 log.cc:1199] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d4bd19f9578f4fa19a6daef427e4a6c8
I20250629 01:51:41.742764 1156 ts_tablet_manager.cc:1939] T d4bd19f9578f4fa19a6daef427e4a6c8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.744130 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d4bd19f9578f4fa19a6daef427e4a6c8 (table default.Table_c586a09ba29642f394d315d9b8ae758c [id=cc648249ae3c443e91004add091222de]) successfully deleted
I20250629 01:51:41.744521 1156 tablet_replica.cc:331] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.744827 1156 raft_consensus.cc:2241] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.745102 1158 tablet_replica.cc:331] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.745131 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_30467a20b5b54ca7894c405b22705bc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:41.745615 1158 raft_consensus.cc:2241] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:41.745741 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30467a20b5b54ca7894c405b22705bc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.746398 1156 raft_consensus.cc:2270] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.746605 1158 raft_consensus.cc:2270] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.747222 1156 ts_tablet_manager.cc:1905] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.747663 1158 ts_tablet_manager.cc:1905] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.749073 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7327f2ea0b414565953b7c1640bc1b59" } new_table_name: "default.Table_2c253a7938ff45d9aeb8a8d6d9324983" modify_external_catalogs: true
I20250629 01:51:41.750371 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3ca4f32bc0954a7c92345faedf91b471"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:41.750778 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ca4f32bc0954a7c92345faedf91b471 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:41.751711 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_115462b9624b4cdb8e39a8f38af93416" } new_table_name: "default.Table_6316ef46c0a6445190d1fa69105eacaa" modify_external_catalogs: true
I20250629 01:51:41.757843 1156 ts_tablet_manager.cc:1918] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.761845 1158 ts_tablet_manager.cc:1918] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.762189 1156 log.cc:1199] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/cc260579c0da4ed38d9b0168e469c1b5
W20250629 01:51:41.762270 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f consensus state has no leader
I20250629 01:51:41.762574 1156 ts_tablet_manager.cc:1939] T cc260579c0da4ed38d9b0168e469c1b5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.763475 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet cc260579c0da4ed38d9b0168e469c1b5 (table default.Table_036318a528c94f2fab8d77b241776f70 [id=d2da066ea4344ed4b3847246f6716583]) successfully deleted
I20250629 01:51:41.763705 1158 log.cc:1199] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/cc260579c0da4ed38d9b0168e469c1b5
I20250629 01:51:41.764056 1158 ts_tablet_manager.cc:1939] T cc260579c0da4ed38d9b0168e469c1b5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.764127 448 master_service.cc:948] ReplaceTablet: received request to replace tablet f98bb48081984d23af55d7fa6a8bec7b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.764746 1154 tablet_bootstrap.cc:492] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.764880 1154 ts_tablet_manager.cc:1397] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.016s
I20250629 01:51:41.764911 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet cc260579c0da4ed38d9b0168e469c1b5 (table default.Table_036318a528c94f2fab8d77b241776f70 [id=d2da066ea4344ed4b3847246f6716583]) successfully deleted
I20250629 01:51:41.765146 1156 tablet_replica.cc:331] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.765306 1156 raft_consensus.cc:2241] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.765444 1156 raft_consensus.cc:2270] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.765516 1154 raft_consensus.cc:357] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.765688 1154 raft_consensus.cc:383] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.765754 1154 raft_consensus.cc:738] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.765846 448 master_service.cc:948] ReplaceTablet: received request to replace tablet caf5826b7e494afc9ae884fe7e94353a from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:41.766024 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 36fcb819c61c4fabbb0da295d0002998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36fcb819c61c4fabbb0da295d0002998 already in progress: creating tablet
I20250629 01:51:41.765974 1154 consensus_queue.cc:260] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.766757 1154 ts_tablet_manager.cc:1428] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:41.767021 1154 tablet_bootstrap.cc:492] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.768365 1154 tablet_bootstrap.cc:654] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.771548 1158 tablet_replica.cc:331] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.771776 1158 raft_consensus.cc:2241] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.771931 1158 raft_consensus.cc:2270] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.772542 1156 ts_tablet_manager.cc:1905] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.772559 1158 ts_tablet_manager.cc:1905] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.778788 1158 ts_tablet_manager.cc:1918] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.778903 1158 log.cc:1199] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d4bd19f9578f4fa19a6daef427e4a6c8
I20250629 01:51:41.779182 1158 ts_tablet_manager.cc:1939] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.779340 447 master_service.cc:948] ReplaceTablet: received request to replace tablet cade2a7984a3413a9690af48b8c28e84 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:41.779665 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d4bd19f9578f4fa19a6daef427e4a6c8 (table default.Table_c586a09ba29642f394d315d9b8ae758c [id=cc648249ae3c443e91004add091222de]) successfully deleted
I20250629 01:51:41.780369 1154 tablet_bootstrap.cc:492] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.780472 1154 ts_tablet_manager.cc:1397] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250629 01:51:41.777802 1156 ts_tablet_manager.cc:1918] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.780715 1156 log.cc:1199] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0c28ea445cec4b35be800dbc8c37af7c
I20250629 01:51:41.781024 1156 ts_tablet_manager.cc:1939] T 0c28ea445cec4b35be800dbc8c37af7c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.781096 1154 raft_consensus.cc:357] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.781244 1154 raft_consensus.cc:383] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.781320 1154 raft_consensus.cc:738] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.781505 1154 consensus_queue.cc:260] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.781800 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0c28ea445cec4b35be800dbc8c37af7c (table default.Table_7a50e6b289ed401092480b4d315d2840 [id=c1e051380c514aa59fd3d4d84ee465d4]) successfully deleted
I20250629 01:51:41.782415 1154 ts_tablet_manager.cc:1428] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:41.782644 1154 tablet_bootstrap.cc:492] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.782883 1156 tablet_replica.cc:331] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.783046 1156 raft_consensus.cc:2241] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.783198 1156 raft_consensus.cc:2270] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.784436 1156 ts_tablet_manager.cc:1905] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.784826 1154 tablet_bootstrap.cc:654] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:41.785236 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfa48f94ac0e48e39578a6e8af5c8e9c consensus state has no leader
W20250629 01:51:41.785984 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7748846a3df14ffa801321ccde86e35f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7748846a3df14ffa801321ccde86e35f already in progress: creating tablet
W20250629 01:51:41.786510 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 881461aa1bc34dd483f65e171c341bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 881461aa1bc34dd483f65e171c341bc5 already in progress: creating tablet
W20250629 01:51:41.786922 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 79e61d9576f8414ebf0074d57cdfb1f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e61d9576f8414ebf0074d57cdfb1f5 already in progress: creating tablet
I20250629 01:51:41.787159 1158 tablet_replica.cc:331] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:41.787305 1158 raft_consensus.cc:2241] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.787449 1158 raft_consensus.cc:2270] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.788081 1158 ts_tablet_manager.cc:1905] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.792675 1156 ts_tablet_manager.cc:1918] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.792778 1156 log.cc:1199] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d550a0ce24e24055947c8344c79bcca3
I20250629 01:51:41.793073 1156 ts_tablet_manager.cc:1939] T d550a0ce24e24055947c8344c79bcca3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.793744 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:41.793825 1158 ts_tablet_manager.cc:1918] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.793912 1158 log.cc:1199] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d550a0ce24e24055947c8344c79bcca3
I20250629 01:51:41.794173 1158 ts_tablet_manager.cc:1939] T d550a0ce24e24055947c8344c79bcca3 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:41.794889 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d550a0ce24e24055947c8344c79bcca3 (table default.Table_daa3abe221c44551848996265764f670 [id=5a1fd538766d4b189cb5dd04bf0b2314]) successfully deleted
I20250629 01:51:41.795604 1154 tablet_bootstrap.cc:492] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.795725 1154 ts_tablet_manager.cc:1397] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20250629 01:51:41.796334 1154 raft_consensus.cc:357] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.796486 1154 raft_consensus.cc:383] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.796540 1154 raft_consensus.cc:738] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.796728 1154 consensus_queue.cc:260] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.797111 1154 ts_tablet_manager.cc:1428] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:41.797353 1154 tablet_bootstrap.cc:492] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.799005 1154 tablet_bootstrap.cc:654] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:41.801630 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 96324ddf32a1460b8fc37e2c324649e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96324ddf32a1460b8fc37e2c324649e6 already in progress: creating tablet
I20250629 01:51:41.803560 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d550a0ce24e24055947c8344c79bcca3 (table default.Table_daa3abe221c44551848996265764f670 [id=5a1fd538766d4b189cb5dd04bf0b2314]) successfully deleted
I20250629 01:51:41.803802 1156 tablet_replica.cc:331] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.803958 1156 raft_consensus.cc:2241] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.803964 1154 tablet_bootstrap.cc:492] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:41.804105 1154 ts_tablet_manager.cc:1397] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20250629 01:51:41.804124 1156 raft_consensus.cc:2270] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.804785 1154 raft_consensus.cc:357] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.804894 1156 ts_tablet_manager.cc:1905] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.804952 1154 raft_consensus.cc:383] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.805043 1154 raft_consensus.cc:738] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.805230 1154 consensus_queue.cc:260] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.805655 1154 ts_tablet_manager.cc:1428] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:41.805929 1154 tablet_bootstrap.cc:492] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:41.807932 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 299922fa5e694bdf96f29f1522068656
I20250629 01:51:41.807964 1154 tablet_bootstrap.cc:654] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.809203 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:41.809458 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:41.809803 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 299922fa5e694bdf96f29f1522068656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 299922fa5e694bdf96f29f1522068656 already in progress: creating tablet
I20250629 01:51:41.810154 1156 ts_tablet_manager.cc:1918] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.810281 1156 log.cc:1199] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ac26cba8a92945dd9acd10e8a76172ca
I20250629 01:51:41.810580 1156 ts_tablet_manager.cc:1939] T ac26cba8a92945dd9acd10e8a76172ca P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.811653 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]). Expected version 1 got 0
I20250629 01:51:41.811931 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6]). Expected version 2 got 0
I20250629 01:51:41.812685 1156 tablet_replica.cc:331] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:41.812826 1156 raft_consensus.cc:2241] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.812844 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_bba7ed21c41c46ca8f68ce01c5225b1e [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 2 got 0
I20250629 01:51:41.812980 1156 raft_consensus.cc:2270] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.813385 1156 ts_tablet_manager.cc:1905] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.815274 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ac26cba8a92945dd9acd10e8a76172ca (table default.Table_abbad35fa11f4103ad64ef1ea77dbd65 [id=29e100c61aaf4178b931f15a4467fc63]) successfully deleted
I20250629 01:51:41.689440 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 7748846a3df14ffa801321ccde86e35f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.689640 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 881461aa1bc34dd483f65e171c341bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.689759 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 with delete_type TABLET_DATA_DELETED (replaced by tablet e82b1352bb4d49769808dc402c85890f) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.711540 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.743279 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d4bd19f9578f4fa19a6daef427e4a6c8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:41.817353 757 tablet_service.cc:1468] Processing CreateTablet for tablet cf4b52fae47f45ad9d77b0a97a854860 (DEFAULT_TABLE table=default.Table_ca28c370cb5e435b9dc8db3efb233a44 [id=53d17bca03a74467a681cb07cb85ef23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.817745 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf4b52fae47f45ad9d77b0a97a854860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.818416 1156 ts_tablet_manager.cc:1918] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.818519 1156 log.cc:1199] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/95c14dacdd0d4156ba7b172edf255ae3
I20250629 01:51:41.818840 1156 ts_tablet_manager.cc:1939] T 95c14dacdd0d4156ba7b172edf255ae3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:41.820694 748 tablet_service.cc:1468] Processing CreateTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a (DEFAULT_TABLE table=default.Table_8b976eb77218429b9513d370b45cd5c9 [id=67701e562ee84eaf9030d915a726ebe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.821030 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28eaa55b3b564e2e9cf698fda98a4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.821349 1155 tablet_bootstrap.cc:492] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:41.821467 1159 tablet_bootstrap.cc:492] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:41.821477 1155 ts_tablet_manager.cc:1397] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.171s user 0.050s sys 0.038s
I20250629 01:51:41.821592 1159 ts_tablet_manager.cc:1397] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.160s user 0.001s sys 0.001s
I20250629 01:51:41.822199 1159 raft_consensus.cc:357] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.822238 1155 raft_consensus.cc:357] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.822348 1159 raft_consensus.cc:383] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.822398 1159 raft_consensus.cc:738] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.822399 1155 raft_consensus.cc:383] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:41.822480 1155 raft_consensus.cc:738] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:41.822571 1159 consensus_queue.cc:260] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.822664 1155 consensus_queue.cc:260] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.823091 1155 ts_tablet_manager.cc:1428] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:41.790227 1227 raft_consensus.cc:491] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.823721 1155 tablet_bootstrap.cc:492] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:41.823696 1227 raft_consensus.cc:513] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.824208 1227 leader_election.cc:290] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.824651 756 tablet_service.cc:1468] Processing CreateTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 (DEFAULT_TABLE table=default.Table_f2f4ee457ae9447697c7e33b84157bd7 [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.824914 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90b206d58d04acdb1a8cb0c45d68c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.810814 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:41.828960 988 leader_election.cc:343] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.829113 988 leader_election.cc:304] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:41.830910 746 tablet_service.cc:1468] Processing CreateTablet for tablet dd709b557b334434a6c2c4a60a996342 (DEFAULT_TABLE table=default.Table_b5f98a84fe7341ac8e6b11b04066323c [id=dd8b9804d0144a879c539614acde7240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.832607 890 tablet_service.cc:1468] Processing CreateTablet for tablet dd709b557b334434a6c2c4a60a996342 (DEFAULT_TABLE table=default.Table_b5f98a84fe7341ac8e6b11b04066323c [id=dd8b9804d0144a879c539614acde7240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.833035 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd709b557b334434a6c2c4a60a996342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.833981 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:41.834237 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]). Expected version 1 got 0
I20250629 01:51:41.837185 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 95c14dacdd0d4156ba7b172edf255ae3 (table default.Table_7724653350cb485cb25696194e364d3f [id=affc7a0fc0c44e73bc261978f6782024]) successfully deleted
W20250629 01:51:41.837461 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 96324ddf32a1460b8fc37e2c324649e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96324ddf32a1460b8fc37e2c324649e6 already in progress: creating tablet
W20250629 01:51:41.837684 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 299922fa5e694bdf96f29f1522068656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 299922fa5e694bdf96f29f1522068656 already in progress: creating tablet
W20250629 01:51:41.837986 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_921f95200675447782407805dfd9fd4a [id=9ff841a004c344c688f5abf6b59b5dce], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet a2562570cb0846e19ebc97bdb43bd70d consensus state has no leader
W20250629 01:51:41.838191 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7748846a3df14ffa801321ccde86e35f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7748846a3df14ffa801321ccde86e35f already in progress: creating tablet
W20250629 01:51:41.838400 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 881461aa1bc34dd483f65e171c341bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 881461aa1bc34dd483f65e171c341bc5 already in progress: creating tablet
W20250629 01:51:41.838586 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 79e61d9576f8414ebf0074d57cdfb1f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79e61d9576f8414ebf0074d57cdfb1f5 already in progress: creating tablet
W20250629 01:51:41.838794 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 96324ddf32a1460b8fc37e2c324649e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96324ddf32a1460b8fc37e2c324649e6 already in progress: creating tablet
W20250629 01:51:41.838994 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 299922fa5e694bdf96f29f1522068656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 299922fa5e694bdf96f29f1522068656 already in progress: creating tablet
I20250629 01:51:41.840341 893 tablet_service.cc:1468] Processing CreateTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a (DEFAULT_TABLE table=default.Table_8b976eb77218429b9513d370b45cd5c9 [id=67701e562ee84eaf9030d915a726ebe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.840668 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28eaa55b3b564e2e9cf698fda98a4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.840878 887 tablet_service.cc:1468] Processing CreateTablet for tablet 0c712d978a3848b9bb3abdbce8314dec (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.841198 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c712d978a3848b9bb3abdbce8314dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.841413 896 tablet_service.cc:1468] Processing CreateTablet for tablet 8a60215417c34bedafbf12172eab469a (DEFAULT_TABLE table=default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.841878 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a60215417c34bedafbf12172eab469a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.844321 888 tablet_service.cc:1468] Processing CreateTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 (DEFAULT_TABLE table=default.Table_f2f4ee457ae9447697c7e33b84157bd7 [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.844650 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90b206d58d04acdb1a8cb0c45d68c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.844520 884 tablet_service.cc:1468] Processing CreateTablet for tablet 0a574f95ac2a4193a563e63cbe120768 (DEFAULT_TABLE table=default.Table_399e960d59fe4143843457bd372dbf63 [id=99712a47e4424c78bb4721eeec6e90c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.845012 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a574f95ac2a4193a563e63cbe120768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.846933 892 tablet_service.cc:1468] Processing CreateTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a (DEFAULT_TABLE table=default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.847215 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91fa0f049904a1fb5cbd38d1fd1693a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.848474 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 (DEFAULT_TABLE table=default.Table_ace97daf931b45ac9676c39144d2931f [id=1669d2f9d2b842bfb8fdbc03b965669a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.848804 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed143377709465d9ae5df7c2ec3ca47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.849180 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_0019f37bf9d34f969b4ef89adc5dfb9d [id=60094a0a7c134476a2a4e44d8555c5d6] alter complete (version 2)
I20250629 01:51:41.852387 894 tablet_service.cc:1468] Processing CreateTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 (DEFAULT_TABLE table=default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 [id=fd2351d583834130bb8213424bf96abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.852753 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69980d5bf37d4001a6196b22c34ccc94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.853726 886 tablet_service.cc:1468] Processing CreateTablet for tablet a32636c293bc44858da7f29a515c467b (DEFAULT_TABLE table=default.Table_2fc84dd48d464ce682425fbd41d92b7e [id=64b627ee531b41ffa46d4bbd813c5b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.854043 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32636c293bc44858da7f29a515c467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.854775 887 tablet_service.cc:1468] Processing CreateTablet for tablet 8976cca816b442cc9a7cb4f23cc13016 (DEFAULT_TABLE table=default.Table_175e2bb2b6c5433da19613f5f2e15823 [id=3fb731afe0f54c88aa9a3643f46e4b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.855098 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8976cca816b442cc9a7cb4f23cc13016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.856333 891 tablet_service.cc:1468] Processing CreateTablet for tablet 351f4a7ae7e244999dda81d282de505d (DEFAULT_TABLE table=default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.856622 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351f4a7ae7e244999dda81d282de505d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.856642 1227 raft_consensus.cc:491] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.856833 1227 raft_consensus.cc:513] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.857523 1227 leader_election.cc:290] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:41.860669 1155 tablet_bootstrap.cc:654] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:41.861363 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 (DEFAULT_TABLE table=default.Table_ace97daf931b45ac9676c39144d2931f [id=1669d2f9d2b842bfb8fdbc03b965669a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.839988 895 tablet_service.cc:1468] Processing CreateTablet for tablet cf4b52fae47f45ad9d77b0a97a854860 (DEFAULT_TABLE table=default.Table_ca28c370cb5e435b9dc8db3efb233a44 [id=53d17bca03a74467a681cb07cb85ef23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.863570 751 tablet_service.cc:1468] Processing CreateTablet for tablet a32636c293bc44858da7f29a515c467b (DEFAULT_TABLE table=default.Table_2fc84dd48d464ce682425fbd41d92b7e [id=64b627ee531b41ffa46d4bbd813c5b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.863843 745 tablet_service.cc:1468] Processing CreateTablet for tablet 8a60215417c34bedafbf12172eab469a (DEFAULT_TABLE table=default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.864082 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4562bba09498449fbff9b7c60bad58fc" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.864414 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d44151acb0d4bf4a630c086b1ac8ff7" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.865415 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet caf5826b7e494afc9ae884fe7e94353a
I20250629 01:51:41.865903 448 catalog_manager.cc:6652] ReplaceTablet: tablet caf5826b7e494afc9ae884fe7e94353a deleted and replaced by tablet 81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:41.867307 744 tablet_service.cc:1468] Processing CreateTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 (DEFAULT_TABLE table=default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 [id=fd2351d583834130bb8213424bf96abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.867105 777 raft_consensus.cc:2466] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:41.869676 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.870754 752 tablet_service.cc:1468] Processing CreateTablet for tablet 0a574f95ac2a4193a563e63cbe120768 (DEFAULT_TABLE table=default.Table_399e960d59fe4143843457bd372dbf63 [id=99712a47e4424c78bb4721eeec6e90c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.872452 889 tablet_service.cc:1468] Processing CreateTablet for tablet 399f460962134ee0b018e9cb8f5d6d2b (DEFAULT_TABLE table=default.Table_de085cd6f5f84a5eabb5f1d77be715b1 [id=282409bfc86b422ba299a651ae80476f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.872658 743 tablet_service.cc:1468] Processing CreateTablet for tablet 399f460962134ee0b018e9cb8f5d6d2b (DEFAULT_TABLE table=default.Table_de085cd6f5f84a5eabb5f1d77be715b1 [id=282409bfc86b422ba299a651ae80476f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.872731 1157 tablet_replica.cc:331] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.848814 882 tablet_service.cc:1468] Processing CreateTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 (DEFAULT_TABLE table=default.Table_f10d54f76b544b88a632ddf72db73ad7 [id=b9df439e13104b14be7c1b522b04ec13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.873346 747 tablet_service.cc:1468] Processing CreateTablet for tablet 351f4a7ae7e244999dda81d282de505d (DEFAULT_TABLE table=default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.873472 1157 raft_consensus.cc:2241] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.873499 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.873680 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf4b52fae47f45ad9d77b0a97a854860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.873919 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0c712d978a3848b9bb3abdbce8314dec (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.874545 1157 raft_consensus.cc:2270] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.874652 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd709b557b334434a6c2c4a60a996342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.876803 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 399f460962134ee0b018e9cb8f5d6d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.877187 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d85fd3d3dd4b07a812c019e1a5c748. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.877525 741 tablet_service.cc:1468] Processing CreateTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 (DEFAULT_TABLE table=default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.880725 1157 ts_tablet_manager.cc:1905] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:41.880815 750 tablet_service.cc:1468] Processing CreateTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 (DEFAULT_TABLE table=default.Table_f10d54f76b544b88a632ddf72db73ad7 [id=b9df439e13104b14be7c1b522b04ec13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.881057 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c712d978a3848b9bb3abdbce8314dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.882498 749 tablet_service.cc:1515] Processing DeleteTablet for tablet caf5826b7e494afc9ae884fe7e94353a with delete_type TABLET_DATA_DELETED (replaced by tablet 81b32bac5a2144d1943285f5f446d53d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:41.880712 754 tablet_service.cc:1468] Processing CreateTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a (DEFAULT_TABLE table=default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:41.883087 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet caf5826b7e494afc9ae884fe7e94353a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf5826b7e494afc9ae884fe7e94353a already in progress: creating tablet
I20250629 01:51:41.883831 883 tablet_service.cc:1468] Processing CreateTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 (DEFAULT_TABLE table=default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.884351 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d85fd3d3dd4b07a812c019e1a5c748. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.884446 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8e0946a20340b38835c4d8daeaa7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.882822 742 tablet_service.cc:1468] Processing CreateTablet for tablet 8976cca816b442cc9a7cb4f23cc13016 (DEFAULT_TABLE table=default.Table_175e2bb2b6c5433da19613f5f2e15823 [id=3fb731afe0f54c88aa9a3643f46e4b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:41.888115 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91fa0f049904a1fb5cbd38d1fd1693a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:41.892712 850 leader_election.cc:343] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:41.899595 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a60215417c34bedafbf12172eab469a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.902058 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32636c293bc44858da7f29a515c467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.902539 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed143377709465d9ae5df7c2ec3ca47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.902835 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 399f460962134ee0b018e9cb8f5d6d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.904583 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351f4a7ae7e244999dda81d282de505d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.905808 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a574f95ac2a4193a563e63cbe120768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.906345 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69980d5bf37d4001a6196b22c34ccc94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.908824 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8e0946a20340b38835c4d8daeaa7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:41.911047 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8976cca816b442cc9a7cb4f23cc13016. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:41.911239 440 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer f304077528b34c61857a659e5873631d (127.0.104.60:32789): Couldn't send request to peer f304077528b34c61857a659e5873631d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250629 01:51:41.938499 1157 ts_tablet_manager.cc:1918] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:41.938627 1157 log.cc:1199] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:41.938980 1157 ts_tablet_manager.cc:1939] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:41.939461 1157 tablet_replica.cc:331] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:41.939661 1157 raft_consensus.cc:2241] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:41.939815 1157 raft_consensus.cc:2270] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:41.934908 850 leader_election.cc:304] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.942591 882 tablet_service.cc:1515] Processing DeleteTablet for tablet caf5826b7e494afc9ae884fe7e94353a with delete_type TABLET_DATA_DELETED (replaced by tablet 81b32bac5a2144d1943285f5f446d53d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:41.950868 1227 raft_consensus.cc:2802] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:41.950982 1227 raft_consensus.cc:491] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:41.951109 1227 raft_consensus.cc:3058] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.953766 1227 raft_consensus.cc:513] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:41.954424 1227 leader_election.cc:290] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:41.955777 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d44151acb0d4bf4a630c086b1ac8ff7" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:41.955978 777 raft_consensus.cc:3058] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.957209 1227 raft_consensus.cc:491] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:41.957361 1227 raft_consensus.cc:513] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.957886 1227 leader_election.cc:290] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:41.958467 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72c6040421642578fac307ea9914ef6" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:41.958642 774 raft_consensus.cc:2466] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:41.959050 850 leader_election.cc:304] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.959312 1227 raft_consensus.cc:2802] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:41.959386 1227 raft_consensus.cc:491] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:41.959458 1227 raft_consensus.cc:3058] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.961575 1227 raft_consensus.cc:513] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.962102 1227 leader_election.cc:290] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:41.962841 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72c6040421642578fac307ea9914ef6" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:41.962997 774 raft_consensus.cc:3058] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:41.965879 774 raft_consensus.cc:2466] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:41.973318 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4562bba09498449fbff9b7c60bad58fc (table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]) successfully deleted
I20250629 01:51:41.974701 850 leader_election.cc:304] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:41.974925 1227 raft_consensus.cc:2802] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:41.975085 1227 raft_consensus.cc:695] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:41.975320 1227 consensus_queue.cc:237] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:41.976929 777 raft_consensus.cc:2466] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:42.000741 1157 ts_tablet_manager.cc:1905] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.009755 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_76c73b5d51504e688e3d59a43f6ab0a0" } new_table_name: "default.Table_d5a40734fce446ae9ad039d5928a7977" modify_external_catalogs: true
I20250629 01:51:42.011495 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f10d54f76b544b88a632ddf72db73ad7" } new_table_name: "default.Table_836f6d2993e14d3d85372201bb2cf8f2" modify_external_catalogs: true
I20250629 01:51:42.012797 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4e8a8a6b9cde425c8326fd3609daf30f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.013422 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e8a8a6b9cde425c8326fd3609daf30f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.014675 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2fc84dd48d464ce682425fbd41d92b7e" } modify_external_catalogs: true
I20250629 01:51:42.014881 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2fc84dd48d464ce682425fbd41d92b7e" } modify_external_catalogs: true
I20250629 01:51:42.016328 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5d35d0525f924ed692497dbaf2287e8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.017588 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d35d0525f924ed692497dbaf2287e8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.018795 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af7bb496140a4999b9999124452b40cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.019282 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af7bb496140a4999b9999124452b40cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.020475 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0735cd5003d948d68959b946189deca4" } modify_external_catalogs: true
I20250629 01:51:42.020637 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0735cd5003d948d68959b946189deca4" } modify_external_catalogs: true
I20250629 01:51:42.021795 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e388d79f0a9c498fb0d0d819be320f34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.022272 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e388d79f0a9c498fb0d0d819be320f34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:42.025035 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 51360us
W20250629 01:51:42.024904 1277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:42.025259 1277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:42.025473 1277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250629 01:51:42.026847 850 leader_election.cc:304] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:42.027148 1227 raft_consensus.cc:2802] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:42.027261 1227 raft_consensus.cc:695] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:42.027501 1227 consensus_queue.cc:237] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.032060 1157 ts_tablet_manager.cc:1918] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.032207 1157 log.cc:1199] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/431dc961cdcf4cd38b521d8d8fc6f406
I20250629 01:51:42.032586 1157 ts_tablet_manager.cc:1939] T 431dc961cdcf4cd38b521d8d8fc6f406 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:42.039124 1277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250629 01:51:42.039278 1277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:42.039414 1277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250629 01:51:42.039527 1277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250629 01:51:42.058044 1277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=34121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
I20250629 01:51:42.058272 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 431dc961cdcf4cd38b521d8d8fc6f406 (table default.Table_a37f4562238b452abf51e6e3d547ca36 [id=c6a48bb9a6874c40b2f7679f2db9e756]) successfully deleted
W20250629 01:51:42.058538 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet caf5826b7e494afc9ae884fe7e94353a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf5826b7e494afc9ae884fe7e94353a already in progress: creating tablet
I20250629 01:51:42.060631 1277 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:42.063239 1277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250629 01:51:42.066907 988 raft_consensus.cc:2684] T d4bd19f9578f4fa19a6daef427e4a6c8 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:42.067152 1311 raft_consensus.cc:491] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.067265 1311 raft_consensus.cc:513] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.067936 1311 leader_election.cc:290] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:42.073288 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a5f38231a34efaab177a23cf693d51" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:42.075754 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:42.080025 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.080049 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.080641 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4562bba09498449fbff9b7c60bad58fc" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:42.080680 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d91fa0f049904a1fb5cbd38d1fd1693a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d91fa0f049904a1fb5cbd38d1fd1693a already in progress: creating tablet
I20250629 01:51:42.080951 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 (DEFAULT_TABLE table=default.Table_ace97daf931b45ac9676c39144d2931f [id=1669d2f9d2b842bfb8fdbc03b965669a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:42.081321 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 246549us
I20250629 01:51:42.081830 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d44151acb0d4bf4a630c086b1ac8ff7" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:42.082077 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d91fa0f049904a1fb5cbd38d1fd1693a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d91fa0f049904a1fb5cbd38d1fd1693a already in progress: creating tablet
I20250629 01:51:42.081135 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 (DEFAULT_TABLE table=default.Table_f2f4ee457ae9447697c7e33b84157bd7 [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.082154 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a5f38231a34efaab177a23cf693d51" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:42.083063 1026 tablet_service.cc:1468] Processing CreateTablet for tablet dd709b557b334434a6c2c4a60a996342 (DEFAULT_TABLE table=default.Table_b5f98a84fe7341ac8e6b11b04066323c [id=dd8b9804d0144a879c539614acde7240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.084002 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_bba7ed21c41c46ca8f68ce01c5225b1e [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 2 got 0
I20250629 01:51:42.085702 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7e0ccdf1ddb349558670c1d0a61d5cf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.086261 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e0ccdf1ddb349558670c1d0a61d5cf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.089072 1155 tablet_bootstrap.cc:492] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.089215 1155 ts_tablet_manager.cc:1397] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.266s user 0.009s sys 0.001s
I20250629 01:51:42.089939 1155 raft_consensus.cc:357] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:42.090097 1155 raft_consensus.cc:383] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.090162 1155 raft_consensus.cc:738] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.090366 1155 consensus_queue.cc:260] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:42.090493 988 leader_election.cc:343] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.090624 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 (DEFAULT_TABLE table=default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 [id=fd2351d583834130bb8213424bf96abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.090688 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4ed143377709465d9ae5df7c2ec3ca47
I20250629 01:51:42.090848 1031 tablet_service.cc:1468] Processing CreateTablet for tablet a32636c293bc44858da7f29a515c467b (DEFAULT_TABLE table=default.Table_2fc84dd48d464ce682425fbd41d92b7e [id=64b627ee531b41ffa46d4bbd813c5b8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.091451 1029 tablet_service.cc:1468] Processing CreateTablet for tablet cf4b52fae47f45ad9d77b0a97a854860 (DEFAULT_TABLE table=default.Table_ca28c370cb5e435b9dc8db3efb233a44 [id=53d17bca03a74467a681cb07cb85ef23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.091977 1155 ts_tablet_manager.cc:1428] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:42.092062 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ed143377709465d9ae5df7c2ec3ca47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.092235 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_04985f57403a42598f1c7ef473177a7b" } new_table_name: "default.Table_fb9fc3beabf04d8d94b0b64c4e12139d" modify_external_catalogs: true
I20250629 01:51:42.092483 1155 tablet_bootstrap.cc:492] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.092617 1154 tablet_bootstrap.cc:492] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.092654 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 399f460962134ee0b018e9cb8f5d6d2b (DEFAULT_TABLE table=default.Table_de085cd6f5f84a5eabb5f1d77be715b1 [id=282409bfc86b422ba299a651ae80476f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.094542 1155 tablet_bootstrap.cc:654] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:42.095194 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f3bf5fd9b674eeb8a5a029152cbae13 consensus state has no leader
I20250629 01:51:42.095902 1159 ts_tablet_manager.cc:1428] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.274s user 0.029s sys 0.059s
I20250629 01:51:42.096135 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_60c45c3633104e8aa039d5d92633c213"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:42.097559 1159 tablet_bootstrap.cc:492] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:42.097651 986 leader_election.cc:343] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.097900 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69980d5bf37d4001a6196b22c34ccc94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.100322 1159 tablet_bootstrap.cc:654] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.100970 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 0a574f95ac2a4193a563e63cbe120768 (DEFAULT_TABLE table=default.Table_399e960d59fe4143843457bd372dbf63 [id=99712a47e4424c78bb4721eeec6e90c3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:42.101044 1360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:42.102039 1358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:42.109997 916 raft_consensus.cc:1273] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:42.110531 1311 consensus_queue.cc:1035] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:51:42.114430 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60c45c3633104e8aa039d5d92633c213 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.116329 1052 raft_consensus.cc:2466] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:42.116355 1051 raft_consensus.cc:2466] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:42.116564 1154 ts_tablet_manager.cc:1397] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.287s user 0.111s sys 0.125s
I20250629 01:51:42.116674 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 351f4a7ae7e244999dda81d282de505d (DEFAULT_TABLE table=default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.117141 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 8a60215417c34bedafbf12172eab469a (DEFAULT_TABLE table=default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.117652 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 0c712d978a3848b9bb3abdbce8314dec (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.118014 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d44151acb0d4bf4a630c086b1ac8ff7" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:42.117722 1154 raft_consensus.cc:357] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.118300 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a (DEFAULT_TABLE table=default.Table_8b976eb77218429b9513d370b45cd5c9 [id=67701e562ee84eaf9030d915a726ebe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.118530 1154 raft_consensus.cc:383] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.118638 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a (DEFAULT_TABLE table=default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.118919 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72c6040421642578fac307ea9914ef6" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:42.118991 1154 raft_consensus.cc:738] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.119115 1048 raft_consensus.cc:2466] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:42.118929 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.118220 1049 raft_consensus.cc:3058] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.119902 1154 consensus_queue.cc:260] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:42.120635 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4ed143377709465d9ae5df7c2ec3ca47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed143377709465d9ae5df7c2ec3ca47 already in progress: creating tablet
I20250629 01:51:42.120981 986 leader_election.cc:304] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:42.121740 1154 ts_tablet_manager.cc:1428] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:42.122116 1311 raft_consensus.cc:2747] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:42.122171 850 leader_election.cc:304] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:42.122363 850 raft_consensus.cc:2684] T 4562bba09498449fbff9b7c60bad58fc P 443c4bd3c5d548ee8cd347d2b4de5293: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:42.122470 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a72c6040421642578fac307ea9914ef6" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:42.122673 1050 raft_consensus.cc:3058] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.122779 1154 tablet_bootstrap.cc:492] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.123553 777 raft_consensus.cc:1273] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:42.126317 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32636c293bc44858da7f29a515c467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.126539 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf4b52fae47f45ad9d77b0a97a854860. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.128170 1022 tablet_service.cc:1468] Processing CreateTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 (DEFAULT_TABLE table=default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.128929 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.129227 1157 tablet_replica.cc:331] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.129204 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 (DEFAULT_TABLE table=default.Table_f10d54f76b544b88a632ddf72db73ad7 [id=b9df439e13104b14be7c1b522b04ec13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.129377 1157 raft_consensus.cc:2241] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.129516 1157 raft_consensus.cc:2270] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.129650 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.130283 1157 ts_tablet_manager.cc:1905] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.130275 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 8976cca816b442cc9a7cb4f23cc13016 (DEFAULT_TABLE table=default.Table_175e2bb2b6c5433da19613f5f2e15823 [id=3fb731afe0f54c88aa9a3643f46e4b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.130715 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a5f38231a34efaab177a23cf693d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.131011 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.131117 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.131237 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet b3de9ab31fc34ba88ac7616dff6e7d5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.131342 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet caf5826b7e494afc9ae884fe7e94353a with delete_type TABLET_DATA_DELETED (replaced by tablet 81b32bac5a2144d1943285f5f446d53d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.131922 1311 consensus_queue.cc:1035] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:42.134130 1158 tablet_replica.cc:331] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.134300 1049 raft_consensus.cc:2466] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:42.134357 1158 raft_consensus.cc:2241] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.134502 1158 raft_consensus.cc:2270] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.135296 1158 ts_tablet_manager.cc:1905] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.143402 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a574f95ac2a4193a563e63cbe120768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.143637 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 399f460962134ee0b018e9cb8f5d6d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.144414 1155 tablet_bootstrap.cc:492] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.144543 1155 ts_tablet_manager.cc:1397] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.002s
I20250629 01:51:42.145169 1155 raft_consensus.cc:357] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:42.145316 1155 raft_consensus.cc:383] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.145383 1155 raft_consensus.cc:738] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.145593 1154 tablet_bootstrap.cc:654] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.145565 1155 consensus_queue.cc:260] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:42.145965 1155 ts_tablet_manager.cc:1428] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:42.146193 1155 tablet_bootstrap.cc:492] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.147624 1155 tablet_bootstrap.cc:654] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.148259 1050 raft_consensus.cc:2466] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:42.148576 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd709b557b334434a6c2c4a60a996342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.148793 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 351f4a7ae7e244999dda81d282de505d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.152299 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8e0946a20340b38835c4d8daeaa7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.156384 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a60215417c34bedafbf12172eab469a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.160243 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90b206d58d04acdb1a8cb0c45d68c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.160985 1154 tablet_bootstrap.cc:492] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.161118 1154 ts_tablet_manager.cc:1397] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.002s
I20250629 01:51:42.161798 1154 raft_consensus.cc:357] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.161955 1154 raft_consensus.cc:383] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.162521 1154 raft_consensus.cc:738] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.162746 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28eaa55b3b564e2e9cf698fda98a4c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.162724 1154 consensus_queue.cc:260] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.163188 1154 ts_tablet_manager.cc:1428] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:42.165521 1154 tablet_bootstrap.cc:492] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.166936 1154 tablet_bootstrap.cc:654] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.167433 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8976cca816b442cc9a7cb4f23cc13016. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.167904 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d91fa0f049904a1fb5cbd38d1fd1693a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.168231 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c712d978a3848b9bb3abdbce8314dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.168975 1155 tablet_bootstrap.cc:492] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.169093 1155 ts_tablet_manager.cc:1397] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.002s
I20250629 01:51:42.169683 1155 raft_consensus.cc:357] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.169834 1155 raft_consensus.cc:383] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.169891 1155 raft_consensus.cc:738] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.170101 1155 consensus_queue.cc:260] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.170490 1155 ts_tablet_manager.cc:1428] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:42.171169 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d85fd3d3dd4b07a812c019e1a5c748. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.171157 1155 tablet_bootstrap.cc:492] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.173576 1155 tablet_bootstrap.cc:654] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.181144 1164 raft_consensus.cc:491] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.181248 1164 raft_consensus.cc:513] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:42.181741 1164 leader_election.cc:290] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:42.182276 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4562bba09498449fbff9b7c60bad58fc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:42.182526 1050 raft_consensus.cc:2466] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:42.182929 710 leader_election.cc:304] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:42.183183 1164 raft_consensus.cc:2802] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:42.183270 1164 raft_consensus.cc:491] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:42.183332 1164 raft_consensus.cc:3058] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.186070 1164 raft_consensus.cc:513] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:42.186745 1164 leader_election.cc:290] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:42.187666 712 leader_election.cc:343] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: 4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:42.190680 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4562bba09498449fbff9b7c60bad58fc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:42.190886 1050 raft_consensus.cc:3058] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:42.194473 712 leader_election.cc:343] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: 4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:42.195641 1050 raft_consensus.cc:2466] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:42.196039 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.196535 710 leader_election.cc:304] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:42.197873 1164 raft_consensus.cc:2802] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:42.198148 1164 raft_consensus.cc:695] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:42.199157 1164 consensus_queue.cc:237] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:42.200896 915 raft_consensus.cc:1273] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:42.201231 1157 ts_tablet_manager.cc:1918] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.201355 1157 log.cc:1199] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d4bd19f9578f4fa19a6daef427e4a6c8
I20250629 01:51:42.201378 1241 consensus_queue.cc:1035] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:42.201720 1157 ts_tablet_manager.cc:1939] T d4bd19f9578f4fa19a6daef427e4a6c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:42.202265 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d4bd19f9578f4fa19a6daef427e4a6c8 (table default.Table_c586a09ba29642f394d315d9b8ae758c [id=cc648249ae3c443e91004add091222de]) successfully deleted
I20250629 01:51:42.209224 1050 raft_consensus.cc:1273] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:42.210633 1241 consensus_queue.cc:1035] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:42.215787 1158 ts_tablet_manager.cc:1918] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.215912 1158 log.cc:1199] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/78a5f38231a34efaab177a23cf693d51
I20250629 01:51:42.216220 1158 ts_tablet_manager.cc:1939] T 78a5f38231a34efaab177a23cf693d51 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.217237 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 78a5f38231a34efaab177a23cf693d51 (table default.Table_601ff9b6b68b4acb9206a64afa4339a8 [id=486534f71b544cee8fd56472d6806f80]) successfully deleted
I20250629 01:51:42.218822 1159 tablet_bootstrap.cc:492] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.219987 1158 tablet_replica.cc:331] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.220216 1159 ts_tablet_manager.cc:1397] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.024s
W20250629 01:51:42.220595 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4ed143377709465d9ae5df7c2ec3ca47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed143377709465d9ae5df7c2ec3ca47 already in progress: creating tablet
W20250629 01:51:42.220752 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4ed143377709465d9ae5df7c2ec3ca47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ed143377709465d9ae5df7c2ec3ca47 already in progress: creating tablet
I20250629 01:51:42.220898 1159 raft_consensus.cc:357] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.221071 1159 raft_consensus.cc:383] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.221712 1159 raft_consensus.cc:738] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.221946 1159 consensus_queue.cc:260] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.222240 1158 raft_consensus.cc:2241] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.222592 1158 raft_consensus.cc:2270] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:42.227541 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26d85fd3d3dd4b07a812c019e1a5c748 consensus state has no leader
I20250629 01:51:42.227922 1158 ts_tablet_manager.cc:1905] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.227185 1159 ts_tablet_manager.cc:1428] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250629 01:51:42.228590 1159 tablet_bootstrap.cc:492] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:42.230729 1159 tablet_bootstrap.cc:654] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.232649 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a32636c293bc44858da7f29a515c467b
I20250629 01:51:42.233685 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a32636c293bc44858da7f29a515c467b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:42.234303 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a32636c293bc44858da7f29a515c467b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a32636c293bc44858da7f29a515c467b already in progress: creating tablet
I20250629 01:51:42.234329 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a32636c293bc44858da7f29a515c467b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:42.234783 1374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250629 01:51:42.235010 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a32636c293bc44858da7f29a515c467b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a32636c293bc44858da7f29a515c467b already in progress: creating tablet
I20250629 01:51:42.239321 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet a32636c293bc44858da7f29a515c467b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.239305 1158 ts_tablet_manager.cc:1918] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.242553 1158 log.cc:1199] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:42.243078 1158 ts_tablet_manager.cc:1939] T 4562bba09498449fbff9b7c60bad58fc P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.245054 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4562bba09498449fbff9b7c60bad58fc (table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]) successfully deleted
I20250629 01:51:42.245577 1158 tablet_replica.cc:331] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.246034 1158 raft_consensus.cc:2241] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.246542 1158 raft_consensus.cc:2270] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.247752 1158 ts_tablet_manager.cc:1905] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.254683 1158 ts_tablet_manager.cc:1918] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.261236 1158 log.cc:1199] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b3de9ab31fc34ba88ac7616dff6e7d5e
I20250629 01:51:42.261785 1158 ts_tablet_manager.cc:1939] T b3de9ab31fc34ba88ac7616dff6e7d5e P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.263135 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b3de9ab31fc34ba88ac7616dff6e7d5e (table default.Table_8603f3907bdd482ea98c45f0c5cd7903 [id=c95a278758c9465da9582afb968ec8f2]) successfully deleted
I20250629 01:51:42.264425 1164 raft_consensus.cc:491] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:42.264443 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet caf5826b7e494afc9ae884fe7e94353a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caf5826b7e494afc9ae884fe7e94353a already in progress: creating tablet
I20250629 01:51:42.264613 1164 raft_consensus.cc:513] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:42.265208 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d91fa0f049904a1fb5cbd38d1fd1693a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d91fa0f049904a1fb5cbd38d1fd1693a already in progress: creating tablet
I20250629 01:51:42.265261 1164 leader_election.cc:290] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.265913 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f40aefcb7e14612bf2d435dfa37d66e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:42.266005 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a32636c293bc44858da7f29a515c467b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a32636c293bc44858da7f29a515c467b already in progress: creating tablet
I20250629 01:51:42.266114 1049 raft_consensus.cc:2466] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:42.266613 710 leader_election.cc:304] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:42.266978 1164 raft_consensus.cc:2802] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:42.267099 1164 raft_consensus.cc:491] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:42.267196 1164 raft_consensus.cc:3058] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.269886 1164 raft_consensus.cc:513] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.270850 1164 leader_election.cc:290] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.271682 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f40aefcb7e14612bf2d435dfa37d66e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:42.272323 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f40aefcb7e14612bf2d435dfa37d66e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:42.272508 1049 raft_consensus.cc:3058] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.274998 1049 raft_consensus.cc:2466] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:42.275336 710 leader_election.cc:304] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:51:42.275671 712 leader_election.cc:343] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.276149 1164 raft_consensus.cc:2802] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:42.276875 1164 raft_consensus.cc:695] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:42.277187 1164 consensus_queue.cc:237] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.278409 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f40aefcb7e14612bf2d435dfa37d66e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:42.278919 712 leader_election.cc:343] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.294345 1339 tablet.cc:1722] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:42.299053 1292 mvcc.cc:204] Tried to move back new op lower bound from 7172759151404863488 to 7172759148002758656. Current Snapshot: MvccSnapshot[applied={T|T < 7172759151404863488}]
I20250629 01:51:42.299688 1382 tablet.cc:1722] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:42.300354 1383 raft_consensus.cc:491] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.300577 1383 raft_consensus.cc:513] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.303041 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6b1ecf098ab4e498391e79cfb1c60b0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:42.303635 988 leader_election.cc:343] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.304555 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6b1ecf098ab4e498391e79cfb1c60b0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:42.305147 986 leader_election.cc:343] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.305302 986 leader_election.cc:304] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:42.308159 1383 leader_election.cc:290] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:42.308634 1385 raft_consensus.cc:2747] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:42.311628 446 master_service.cc:948] ReplaceTablet: received request to replace tablet dd6013bb40c248f29f40eab2c3008f6b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.312991 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c28ea445cec4b35be800dbc8c37af7c from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.313129 1384 tablet.cc:1722] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:51:42.315486 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b4701123dfa467ebb6c95f9742b3874 consensus state has no leader
I20250629 01:51:42.316227 1382 tablet.cc:1722] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:42.316967 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9cf506d4e74e4895b25ee07125d2b234
I20250629 01:51:42.322926 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e605f1f30c8d4fdd8503b71227359798"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.323770 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e605f1f30c8d4fdd8503b71227359798 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.327116 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c5f8b9dc8c7d48eb881b636fd9096392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.327540 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5f8b9dc8c7d48eb881b636fd9096392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.328804 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f641040c5d3f4fabb4642ae6c32b10c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.329218 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f641040c5d3f4fabb4642ae6c32b10c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.329793 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_58446e63991843cea406fca1f041d6ca" } new_table_name: "default.Table_1048b0cf35964b1fb36e7004e009ac0e" modify_external_catalogs: true
I20250629 01:51:42.334762 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ed143377709465d9ae5df7c2ec3ca47 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.337574 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.337769 1156 tablet_replica.cc:331] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.337921 1156 raft_consensus.cc:2241] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:42.338090 1156 pending_rounds.cc:70] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:42.338142 1156 pending_rounds.cc:77] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759151409942528 op_type: NO_OP noop_request { }
I20250629 01:51:42.338223 1156 raft_consensus.cc:2887] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:42.338303 1156 raft_consensus.cc:2270] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.338876 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6e7aa21adf4540bcb0f4c7f4e711f90b" } new_table_name: "default.Table_1961bae4492d41e8b6c8536a9f324977" modify_external_catalogs: true
I20250629 01:51:42.339452 1156 ts_tablet_manager.cc:1905] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.341341 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_adf0dc244a084dcf82f6cb6f22ecc128"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.341818 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adf0dc244a084dcf82f6cb6f22ecc128 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.343410 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.343657 1158 tablet_replica.cc:331] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.343891 1158 raft_consensus.cc:2241] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.344527 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 64477468c6da4b4ba282d8530eb5cbb5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.345010 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb" } modify_external_catalogs: true
I20250629 01:51:42.345140 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb" } modify_external_catalogs: true
I20250629 01:51:42.344066 1158 raft_consensus.cc:2270] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.349506 1158 ts_tablet_manager.cc:1905] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.350610 1156 ts_tablet_manager.cc:1918] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:42.350790 1156 log.cc:1199] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:42.351174 1156 ts_tablet_manager.cc:1939] T 4562bba09498449fbff9b7c60bad58fc P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.351652 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4562bba09498449fbff9b7c60bad58fc (table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]) successfully deleted
I20250629 01:51:42.353055 1277 server_base.cc:1048] running on GCE node
I20250629 01:51:42.354086 1373 mvcc.cc:204] Tried to move back new op lower bound from 7172759151404863488 to 7172759148002758656. Current Snapshot: MvccSnapshot[applied={T|T < 7172759151404863488}]
I20250629 01:51:42.357393 1277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:42.358049 1380 tablet.cc:1722] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:42.358614 1387 tablet.cc:1722] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:42.359848 1155 tablet_bootstrap.cc:492] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.359972 1155 ts_tablet_manager.cc:1397] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.189s user 0.006s sys 0.034s
I20250629 01:51:42.360667 1155 raft_consensus.cc:357] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.360827 1155 raft_consensus.cc:383] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.360915 1155 raft_consensus.cc:738] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.361099 1155 consensus_queue.cc:260] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.365468 1158 ts_tablet_manager.cc:1918] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.365792 1158 log.cc:1199] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/3a45e95a4a6a45f9b7c7b09bd69112b4
I20250629 01:51:42.367496 1277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:42.367642 1158 ts_tablet_manager.cc:1939] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.368580 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 (table default.Table_0b441ce248f2448eabf2d7e3cddff976 [id=ba643eccd0e54ce083459bf535db1fe1]) successfully deleted
I20250629 01:51:42.368813 1277 hybrid_clock.cc:648] HybridClock initialized: now 1751161902368742 us; error 59 us; skew 500 ppm
I20250629 01:51:42.369204 1277 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:42.361593 1155 ts_tablet_manager.cc:1428] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:42.370415 1155 tablet_bootstrap.cc:492] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.371989 1155 tablet_bootstrap.cc:654] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.373548 1155 tablet_bootstrap.cc:492] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.373721 1155 ts_tablet_manager.cc:1397] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250629 01:51:42.373829 1159 tablet_bootstrap.cc:492] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.374253 1155 raft_consensus.cc:357] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.374384 1155 raft_consensus.cc:383] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.374457 1155 raft_consensus.cc:738] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.375072 1154 tablet_bootstrap.cc:492] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.375059 1159 ts_tablet_manager.cc:1397] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.145s user 0.029s sys 0.030s
I20250629 01:51:42.375200 1154 ts_tablet_manager.cc:1397] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.210s user 0.011s sys 0.029s
I20250629 01:51:42.375821 1154 raft_consensus.cc:357] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.375977 1154 raft_consensus.cc:383] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.376039 1154 raft_consensus.cc:738] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.376883 1159 raft_consensus.cc:357] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.376868 1154 consensus_queue.cc:260] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.377362 1154 ts_tablet_manager.cc:1428] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:42.377682 1154 tablet_bootstrap.cc:492] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.379122 1154 tablet_bootstrap.cc:654] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.379878 1155 consensus_queue.cc:260] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.380234 1159 raft_consensus.cc:383] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.380317 1159 raft_consensus.cc:738] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:51:42.380985 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 543242e4a5064268ae1e5519b569f682 consensus state has no leader
I20250629 01:51:42.381202 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 5f3bf5fd9b674eeb8a5a029152cbae13 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.381173 1159 consensus_queue.cc:260] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.381434 1155 ts_tablet_manager.cc:1428] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250629 01:51:42.381601 1159 ts_tablet_manager.cc:1428] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250629 01:51:42.381933 1155 tablet_bootstrap.cc:492] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.382304 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]). Expected version 2 got 0
I20250629 01:51:42.382539 451 catalog_manager.cc:5582] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:42.382928 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]). Expected version 1 got 0
I20250629 01:51:42.383152 451 catalog_manager.cc:5582] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:42.383453 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_bba7ed21c41c46ca8f68ce01c5225b1e [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 2 got 0
I20250629 01:51:42.383723 1155 tablet_bootstrap.cc:654] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.384902 755 tablet_service.cc:1468] Processing CreateTablet for tablet 9624117e97974a47ac2bfe525da6f405 (DEFAULT_TABLE table=default.Table_3906fb981a8640e1906a78a321c479b2 [id=e2a731484fe346738ed611d5d7312092]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.385290 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9624117e97974a47ac2bfe525da6f405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.386092 1159 tablet_bootstrap.cc:492] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:42.386958 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9624117e97974a47ac2bfe525da6f405 (DEFAULT_TABLE table=default.Table_3906fb981a8640e1906a78a321c479b2 [id=e2a731484fe346738ed611d5d7312092]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.387228 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9624117e97974a47ac2bfe525da6f405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.388372 1155 tablet_bootstrap.cc:492] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.388345 746 tablet_service.cc:1468] Processing CreateTablet for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (DEFAULT_TABLE table=default.Table_c6543e800e6f41d4ac45569c0650a8f4 [id=6ce2537a6faf4e11ab48c7fafc6451e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.388468 1155 ts_tablet_manager.cc:1397] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20250629 01:51:42.388648 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.390602 1155 raft_consensus.cc:357] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.390621 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (DEFAULT_TABLE table=default.Table_c6543e800e6f41d4ac45569c0650a8f4 [id=6ce2537a6faf4e11ab48c7fafc6451e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.390763 1155 raft_consensus.cc:383] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.390846 1155 raft_consensus.cc:738] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.390990 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.391034 1155 consensus_queue.cc:260] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.391659 1154 tablet_bootstrap.cc:492] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.391801 1154 ts_tablet_manager.cc:1397] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20250629 01:51:42.392544 1154 raft_consensus.cc:357] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.392701 1154 raft_consensus.cc:383] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.392778 1154 raft_consensus.cc:738] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.392968 1154 consensus_queue.cc:260] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.393348 1154 ts_tablet_manager.cc:1428] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:42.393342 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_becfa32b77bb40b1b9df61126ffe628e" } modify_external_catalogs: true
I20250629 01:51:42.393476 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_becfa32b77bb40b1b9df61126ffe628e" } modify_external_catalogs: true
I20250629 01:51:42.394129 1154 tablet_bootstrap.cc:492] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.394814 1159 tablet_bootstrap.cc:654] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.394806 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 (DEFAULT_TABLE table=default.Table_3ca4f32bc0954a7c92345faedf91b471 [id=48089fc1034648639f16290cb405456a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.395155 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49333302a27f4ac3b76a1e492b3c1b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.395613 1154 tablet_bootstrap.cc:654] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:42.396426 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82d58c88a6994d828b444c5814b287ff consensus state has no leader
I20250629 01:51:42.396426 1018 tablet_service.cc:1468] Processing CreateTablet for tablet e1fe3880e51345269e852014d0d60bb6 (DEFAULT_TABLE table=default.Table_bf61fc85ef4b405180f3b4621489590e [id=4e38782b406e4cd9ae3278d074a1d97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.396710 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_91eb82c0338648b19d82e9ef4de65851" } new_table_name: "default.Table_8c1fbbaabe004466858cce35b18194b1" modify_external_catalogs: true
I20250629 01:51:42.396816 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1fe3880e51345269e852014d0d60bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.399464 1155 ts_tablet_manager.cc:1428] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250629 01:51:42.399689 1155 tablet_bootstrap.cc:492] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.399693 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 (DEFAULT_TABLE table=default.Table_f0a26d9ea3e8407da5ca0813db26beb3 [id=b52a4204ccf34819b3782aca3ea1db0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.399959 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86832a3c26224c7ea0b9b4b6866d3880. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.400967 1155 tablet_bootstrap.cc:654] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.402601 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9624117e97974a47ac2bfe525da6f405 (DEFAULT_TABLE table=default.Table_3906fb981a8640e1906a78a321c479b2 [id=e2a731484fe346738ed611d5d7312092]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.402966 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9624117e97974a47ac2bfe525da6f405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.403635 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 543242e4a5064268ae1e5519b569f682 (DEFAULT_TABLE table=default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.403990 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543242e4a5064268ae1e5519b569f682. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.408910 1154 tablet_bootstrap.cc:492] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.409045 1154 ts_tablet_manager.cc:1397] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.004s
I20250629 01:51:42.409130 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ae2d1a922b914ba1bfa2dbca0cda5013" } modify_external_catalogs: true
I20250629 01:51:42.409281 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ae2d1a922b914ba1bfa2dbca0cda5013" } modify_external_catalogs: true
I20250629 01:51:42.409649 1154 raft_consensus.cc:357] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.410144 745 tablet_service.cc:1468] Processing CreateTablet for tablet e1fe3880e51345269e852014d0d60bb6 (DEFAULT_TABLE table=default.Table_bf61fc85ef4b405180f3b4621489590e [id=4e38782b406e4cd9ae3278d074a1d97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.410306 1154 raft_consensus.cc:383] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.410377 1154 raft_consensus.cc:738] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.410486 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1fe3880e51345269e852014d0d60bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.410571 1154 consensus_queue.cc:260] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.411595 1154 ts_tablet_manager.cc:1428] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:42.412066 1154 tablet_bootstrap.cc:492] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.412329 741 tablet_service.cc:1468] Processing CreateTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 (DEFAULT_TABLE table=default.Table_f0a26d9ea3e8407da5ca0813db26beb3 [id=b52a4204ccf34819b3782aca3ea1db0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.412602 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86832a3c26224c7ea0b9b4b6866d3880. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.414621 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (DEFAULT_TABLE table=default.Table_c6543e800e6f41d4ac45569c0650a8f4 [id=6ce2537a6faf4e11ab48c7fafc6451e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.414968 895 tablet_service.cc:1468] Processing CreateTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 (DEFAULT_TABLE table=default.Table_3ca4f32bc0954a7c92345faedf91b471 [id=48089fc1034648639f16290cb405456a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.415220 893 tablet_service.cc:1468] Processing CreateTablet for tablet e1fe3880e51345269e852014d0d60bb6 (DEFAULT_TABLE table=default.Table_bf61fc85ef4b405180f3b4621489590e [id=4e38782b406e4cd9ae3278d074a1d97e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.415704 891 tablet_service.cc:1468] Processing CreateTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 (DEFAULT_TABLE table=default.Table_f0a26d9ea3e8407da5ca0813db26beb3 [id=b52a4204ccf34819b3782aca3ea1db0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.415843 1154 tablet_bootstrap.cc:654] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.416213 889 tablet_service.cc:1468] Processing CreateTablet for tablet 543242e4a5064268ae1e5519b569f682 (DEFAULT_TABLE table=default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.414981 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.417593 1154 tablet_bootstrap.cc:492] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.417693 1154 ts_tablet_manager.cc:1397] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:42.417783 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1fe3880e51345269e852014d0d60bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.417735 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]). Expected version 2 got 0
I20250629 01:51:42.418088 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49333302a27f4ac3b76a1e492b3c1b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.420769 744 tablet_service.cc:1468] Processing CreateTablet for tablet 543242e4a5064268ae1e5519b569f682 (DEFAULT_TABLE table=default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.421346 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543242e4a5064268ae1e5519b569f682. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.421449 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 543242e4a5064268ae1e5519b569f682. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.422238 743 tablet_service.cc:1468] Processing CreateTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de (DEFAULT_TABLE table=default.Table_91eb82c0338648b19d82e9ef4de65851 [id=700ef08e094b44369154076e65d9ea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.422545 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0827ec622ff04ca6b2fec50e40b9e1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.426218 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86832a3c26224c7ea0b9b4b6866d3880. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.426713 751 tablet_service.cc:1468] Processing CreateTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 (DEFAULT_TABLE table=default.Table_3ca4f32bc0954a7c92345faedf91b471 [id=48089fc1034648639f16290cb405456a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.427397 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49333302a27f4ac3b76a1e492b3c1b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.428325 888 tablet_service.cc:1468] Processing CreateTablet for tablet e82b1352bb4d49769808dc402c85890f (DEFAULT_TABLE table=default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.428638 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82b1352bb4d49769808dc402c85890f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.430088 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de (DEFAULT_TABLE table=default.Table_91eb82c0338648b19d82e9ef4de65851 [id=700ef08e094b44369154076e65d9ea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.430395 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0827ec622ff04ca6b2fec50e40b9e1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.431568 1155 tablet_bootstrap.cc:492] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.431908 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92433b26d65e4bf6b433995ad26b2a6b" } new_table_name: "default.Table_a43790ad2a5d42c986031de7a39591bf" modify_external_catalogs: true
I20250629 01:51:42.432313 882 tablet_service.cc:1468] Processing CreateTablet for tablet bd5150eff1ad43588a9e70dab47744e3 (DEFAULT_TABLE table=default.Table_a726ca476374467fb904639d799b7a1c [id=c483797822d647458b5728bf3255b15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.432616 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5150eff1ad43588a9e70dab47744e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.433072 1155 ts_tablet_manager.cc:1397] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20250629 01:51:42.433702 1155 raft_consensus.cc:357] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.434597 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e388d79f0a9c498fb0d0d819be320f34" } modify_external_catalogs: true
I20250629 01:51:42.434720 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e388d79f0a9c498fb0d0d819be320f34" } modify_external_catalogs: true
I20250629 01:51:42.434944 893 tablet_service.cc:1468] Processing CreateTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 (DEFAULT_TABLE table=default.Table_30467a20b5b54ca7894c405b22705bc2 [id=66af0a365c3f40deac642473f9125b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.435227 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b680859a1a4c56afe33f5b1b3c0793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.435690 1155 raft_consensus.cc:383] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.435781 1155 raft_consensus.cc:738] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.436174 1155 consensus_queue.cc:260] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.436368 883 tablet_service.cc:1468] Processing CreateTablet for tablet 0b16b6840c24484b90218fffc1da9621 (DEFAULT_TABLE table=default.Table_81bc81a3198343948ffbd72f6d683481 [id=14974743edd741fbab51f9e8a4add096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.436589 1155 ts_tablet_manager.cc:1428] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:42.436654 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b16b6840c24484b90218fffc1da9621. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.437067 752 tablet_service.cc:1468] Processing CreateTablet for tablet e82b1352bb4d49769808dc402c85890f (DEFAULT_TABLE table=default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.437577 1155 tablet_bootstrap.cc:492] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.437794 884 tablet_service.cc:1468] Processing CreateTablet for tablet a792a26725954adea15ad57e31bfb224 (DEFAULT_TABLE table=default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.438112 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a792a26725954adea15ad57e31bfb224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.440268 754 tablet_service.cc:1468] Processing CreateTablet for tablet bd5150eff1ad43588a9e70dab47744e3 (DEFAULT_TABLE table=default.Table_a726ca476374467fb904639d799b7a1c [id=c483797822d647458b5728bf3255b15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.440340 885 tablet_service.cc:1468] Processing CreateTablet for tablet 77eae224763d464496066b4a98efe5ff (DEFAULT_TABLE table=default.Table_6f7d1b03023842daa7f9d43483dc6b95 [id=92be86eee6b64fb0984d8e5ea727fbdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.440656 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77eae224763d464496066b4a98efe5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.440862 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82b1352bb4d49769808dc402c85890f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.442613 1154 raft_consensus.cc:357] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.442775 1154 raft_consensus.cc:383] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.442983 1154 raft_consensus.cc:738] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.443174 1154 consensus_queue.cc:260] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.443861 1154 ts_tablet_manager.cc:1428] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250629 01:51:42.443948 886 tablet_service.cc:1468] Processing CreateTablet for tablet 69f0ae4c51924370896c624affde6761 (DEFAULT_TABLE table=default.Table_becfa32b77bb40b1b9df61126ffe628e [id=1c654e447e1b46f1a4570b41df8b1861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.444267 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0ae4c51924370896c624affde6761. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.444234 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.444645 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.445547 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de (DEFAULT_TABLE table=default.Table_91eb82c0338648b19d82e9ef4de65851 [id=700ef08e094b44369154076e65d9ea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.446244 1277 webserver.cc:469] Webserver started at http://127.0.104.60:34121/ using document root <none> and password file <none>
I20250629 01:51:42.447180 1031 tablet_service.cc:1468] Processing CreateTablet for tablet e82b1352bb4d49769808dc402c85890f (DEFAULT_TABLE table=default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.447368 1277 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:42.447515 1277 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:42.447466 1019 tablet_service.cc:1468] Processing CreateTablet for tablet bd5150eff1ad43588a9e70dab47744e3 (DEFAULT_TABLE table=default.Table_a726ca476374467fb904639d799b7a1c [id=c483797822d647458b5728bf3255b15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.448792 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0827ec622ff04ca6b2fec50e40b9e1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.449035 1158 tablet_replica.cc:331] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.449762 1158 raft_consensus.cc:2241] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.449936 1158 raft_consensus.cc:2270] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.450387 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e82b1352bb4d49769808dc402c85890f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.451192 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5150eff1ad43588a9e70dab47744e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.451936 1154 tablet_bootstrap.cc:492] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.452805 1159 tablet_bootstrap.cc:492] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.452926 1159 ts_tablet_manager.cc:1397] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.003s
I20250629 01:51:42.454151 1159 raft_consensus.cc:357] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.454300 1159 raft_consensus.cc:383] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.454399 1159 raft_consensus.cc:738] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.454643 1159 consensus_queue.cc:260] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.455271 1159 ts_tablet_manager.cc:1428] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:42.457084 1159 tablet_bootstrap.cc:492] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:42.457890 756 tablet_service.cc:1468] Processing CreateTablet for tablet 69f0ae4c51924370896c624affde6761 (DEFAULT_TABLE table=default.Table_becfa32b77bb40b1b9df61126ffe628e [id=1c654e447e1b46f1a4570b41df8b1861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.458880 747 tablet_service.cc:1468] Processing CreateTablet for tablet 77eae224763d464496066b4a98efe5ff (DEFAULT_TABLE table=default.Table_6f7d1b03023842daa7f9d43483dc6b95 [id=92be86eee6b64fb0984d8e5ea727fbdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.459007 1158 ts_tablet_manager.cc:1905] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.459050 750 tablet_service.cc:1468] Processing CreateTablet for tablet a792a26725954adea15ad57e31bfb224 (DEFAULT_TABLE table=default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.459151 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5150eff1ad43588a9e70dab47744e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.459194 757 tablet_service.cc:1468] Processing CreateTablet for tablet 0b16b6840c24484b90218fffc1da9621 (DEFAULT_TABLE table=default.Table_81bc81a3198343948ffbd72f6d683481 [id=14974743edd741fbab51f9e8a4add096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.460259 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77eae224763d464496066b4a98efe5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.461076 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a792a26725954adea15ad57e31bfb224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.461589 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b16b6840c24484b90218fffc1da9621. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.462052 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d61687e19d994e87b09e55bd3b9476d7
I20250629 01:51:42.464274 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet dd6013bb40c248f29f40eab2c3008f6b
I20250629 01:51:42.464637 446 catalog_manager.cc:6652] ReplaceTablet: tablet dd6013bb40c248f29f40eab2c3008f6b deleted and replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:42.464923 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f128352ab4bd4f258d90d9a4022577a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:42.465188 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0ae4c51924370896c624affde6761. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:42.465384 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f128352ab4bd4f258d90d9a4022577a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:42.466398 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0827ec622ff04ca6b2fec50e40b9e1de consensus state has no leader
I20250629 01:51:42.466786 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:42.467402 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5f3bf5fd9b674eeb8a5a029152cbae13
I20250629 01:51:42.467732 449 catalog_manager.cc:6652] ReplaceTablet: tablet 5f3bf5fd9b674eeb8a5a029152cbae13 deleted and replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5
I20250629 01:51:42.469273 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 0abdecb6d38643cc82d46fa9d4453085
I20250629 01:51:42.470150 883 tablet_service.cc:1515] Processing DeleteTablet for tablet dd6013bb40c248f29f40eab2c3008f6b with delete_type TABLET_DATA_DELETED (replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.470675 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 69f0ae4c51924370896c624affde6761
I20250629 01:51:42.470834 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0b16b6840c24484b90218fffc1da9621 (DEFAULT_TABLE table=default.Table_81bc81a3198343948ffbd72f6d683481 [id=14974743edd741fbab51f9e8a4add096]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.474412 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 with delete_type TABLET_DATA_DELETED (replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.476037 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 69f0ae4c51924370896c624affde6761 (DEFAULT_TABLE table=default.Table_becfa32b77bb40b1b9df61126ffe628e [id=1c654e447e1b46f1a4570b41df8b1861]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.477015 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b16b6840c24484b90218fffc1da9621. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.478621 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0ae4c51924370896c624affde6761. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.478716 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f0ae4c51924370896c624affde6761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.478947 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.479048 889 tablet_service.cc:1515] Processing DeleteTablet for tablet d61687e19d994e87b09e55bd3b9476d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.479291 1159 tablet_bootstrap.cc:654] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.479996 1155 tablet_bootstrap.cc:654] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.481487 742 tablet_service.cc:1515] Processing DeleteTablet for tablet d61687e19d994e87b09e55bd3b9476d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.481381 455 catalog_manager.cc:5582] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:42.481696 889 tablet_service.cc:1515] Processing DeleteTablet for tablet d4bd19f9578f4fa19a6daef427e4a6c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.481739 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:42.482887 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 with delete_type TABLET_DATA_DELETED (replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.483359 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a792a26725954adea15ad57e31bfb224 (DEFAULT_TABLE table=default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.483654 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a792a26725954adea15ad57e31bfb224. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:42.484802 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d61687e19d994e87b09e55bd3b9476d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61687e19d994e87b09e55bd3b9476d7 already in progress: creating tablet
W20250629 01:51:42.485119 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f3bf5fd9b674eeb8a5a029152cbae13 already in progress: creating tablet
W20250629 01:51:42.485544 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet dd6013bb40c248f29f40eab2c3008f6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd6013bb40c248f29f40eab2c3008f6b already in progress: creating tablet
W20250629 01:51:42.486160 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 69f0ae4c51924370896c624affde6761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f0ae4c51924370896c624affde6761 already in progress: creating tablet
W20250629 01:51:42.486220 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e508393b9cb044c4b90194c8aab3184a consensus state has no leader
W20250629 01:51:42.486371 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 69980d5bf37d4001a6196b22c34ccc94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69980d5bf37d4001a6196b22c34ccc94 already in progress: creating tablet
W20250629 01:51:42.486665 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d61687e19d994e87b09e55bd3b9476d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61687e19d994e87b09e55bd3b9476d7 already in progress: creating tablet
W20250629 01:51:42.486841 442 catalog_manager.cc:4895] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d4bd19f9578f4fa19a6daef427e4a6c8 because the tablet was not found. No further retry: Not found: Tablet not found: d4bd19f9578f4fa19a6daef427e4a6c8
I20250629 01:51:42.486886 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:42.486985 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f3bf5fd9b674eeb8a5a029152cbae13 already in progress: creating tablet
W20250629 01:51:42.487461 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 69980d5bf37d4001a6196b22c34ccc94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69980d5bf37d4001a6196b22c34ccc94 already in progress: creating tablet
I20250629 01:51:42.487895 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet dd6013bb40c248f29f40eab2c3008f6b with delete_type TABLET_DATA_DELETED (replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.488322 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f0ae4c51924370896c624affde6761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.487846 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d61687e19d994e87b09e55bd3b9476d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:42.488966 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 69f0ae4c51924370896c624affde6761 because the tablet was not found. No further retry: Not found: Tablet not found: 69f0ae4c51924370896c624affde6761
I20250629 01:51:42.489140 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 77eae224763d464496066b4a98efe5ff (DEFAULT_TABLE table=default.Table_6f7d1b03023842daa7f9d43483dc6b95 [id=92be86eee6b64fb0984d8e5ea727fbdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.489430 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77eae224763d464496066b4a98efe5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.490329 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.491151 1241 raft_consensus.cc:491] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.493000 1241 raft_consensus.cc:513] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.493567 1241 leader_election.cc:290] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.494127 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ae96f2da36744198e3d38cb07bb47a0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:42.494741 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f0ae4c51924370896c624affde6761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.496662 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ae96f2da36744198e3d38cb07bb47a0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:42.496721 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 because the tablet was not found. No further retry: Not found: Tablet not found: 3a45e95a4a6a45f9b7c7b09bd69112b4
I20250629 01:51:42.496918 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:42.496672 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 with delete_type TABLET_DATA_DELETED (replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:42.499243 712 leader_election.cc:343] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.499920 747 tablet_service.cc:1515] Processing DeleteTablet for tablet dd6013bb40c248f29f40eab2c3008f6b with delete_type TABLET_DATA_DELETED (replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.500445 1277 fs_manager.cc:714] Time spent opening directory manager: real 0.038s user 0.008s sys 0.014s
W20250629 01:51:42.500749 710 leader_election.cc:343] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.502328 1154 tablet_bootstrap.cc:654] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.503535 1350 raft_consensus.cc:491] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.503660 1350 raft_consensus.cc:513] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.504263 1350 leader_election.cc:290] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.509747 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_58a60b60dab64affa9d8b5b1acf53326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.510234 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58a60b60dab64affa9d8b5b1acf53326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.510500 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f40aefcb7e14612bf2d435dfa37d66e" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:42.510746 1050 raft_consensus.cc:2391] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 443c4bd3c5d548ee8cd347d2b4de5293 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:51:42.511732 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c70f3648ce00471bb5d6dbd7d12e6b50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.512321 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.513432 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8] alter complete (version 2)
I20250629 01:51:42.513656 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e0533c0c40ca4af1be6ac30ee37f3aed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.514094 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0533c0c40ca4af1be6ac30ee37f3aed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.515478 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81] alter complete (version 2)
I20250629 01:51:42.517467 1017 tablet_service.cc:1468] Processing CreateTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 (DEFAULT_TABLE table=default.Table_30467a20b5b54ca7894c405b22705bc2 [id=66af0a365c3f40deac642473f9125b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.518343 891 tablet_service.cc:1515] Processing DeleteTablet for tablet cc260579c0da4ed38d9b0168e469c1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.518579 1157 tablet_replica.cc:331] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.518752 1157 raft_consensus.cc:2241] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.518885 1157 raft_consensus.cc:2270] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.519201 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c6e85ae2e824442fa3e307eda6dc1aa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:42.519428 1157 ts_tablet_manager.cc:1905] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:42.519577 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.520352 1158 ts_tablet_manager.cc:1918] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.523420 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c5f8b9dc8c7d48eb881b636fd9096392" } new_table_name: "default.Table_abc4bc33793d4bee824f97f24a172593" modify_external_catalogs: true
I20250629 01:51:42.524370 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_358f4795c7e244f2b72efe50dacd310f" } modify_external_catalogs: true
I20250629 01:51:42.524504 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_358f4795c7e244f2b72efe50dacd310f" } modify_external_catalogs: true
I20250629 01:51:42.524984 886 tablet_service.cc:1515] Processing DeleteTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 with delete_type TABLET_DATA_DELETED (replaced by tablet 36fcb819c61c4fabbb0da295d0002998) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.525144 710 leader_election.cc:304] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:42.525249 1157 ts_tablet_manager.cc:1918] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.525377 1157 log.cc:1199] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/cc260579c0da4ed38d9b0168e469c1b5
I20250629 01:51:42.525633 1241 raft_consensus.cc:2747] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:42.525683 1157 ts_tablet_manager.cc:1939] T cc260579c0da4ed38d9b0168e469c1b5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:42.525820 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0f40aefcb7e14612bf2d435dfa37d66e" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:42.525827 746 tablet_service.cc:1468] Processing CreateTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 (DEFAULT_TABLE table=default.Table_30467a20b5b54ca7894c405b22705bc2 [id=66af0a365c3f40deac642473f9125b10]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.526713 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b680859a1a4c56afe33f5b1b3c0793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.526856 850 leader_election.cc:304] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:42.527184 1350 raft_consensus.cc:3058] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.529906 1350 raft_consensus.cc:2747] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:42.530246 1158 log.cc:1199] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e6b1ecf098ab4e498391e79cfb1c60b0
I20250629 01:51:42.532217 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet cc260579c0da4ed38d9b0168e469c1b5 (table default.Table_036318a528c94f2fab8d77b241776f70 [id=d2da066ea4344ed4b3847246f6716583]) successfully deleted
I20250629 01:51:42.532521 1157 tablet_replica.cc:331] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.532733 1157 raft_consensus.cc:2241] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.532943 1155 tablet_bootstrap.cc:492] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.532963 1157 raft_consensus.cc:2270] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.533075 1155 ts_tablet_manager.cc:1397] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.001s
I20250629 01:51:42.533668 1157 ts_tablet_manager.cc:1905] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.533777 1155 raft_consensus.cc:357] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.533934 1155 raft_consensus.cc:383] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.534022 1155 raft_consensus.cc:738] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.534212 1155 consensus_queue.cc:260] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.534900 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9b680859a1a4c56afe33f5b1b3c0793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.536230 1155 ts_tablet_manager.cc:1428] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:42.536454 1155 tablet_bootstrap.cc:492] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.538580 1155 tablet_bootstrap.cc:654] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.540000 1158 ts_tablet_manager.cc:1939] T e6b1ecf098ab4e498391e79cfb1c60b0 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.546054 886 tablet_service.cc:1468] Processing CreateTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 (DEFAULT_TABLE table=default.Table_f641040c5d3f4fabb4642ae6c32b10c7 [id=7607f62687244e718732520ce1342b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.546412 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7c1f1af39d47c191d34dc026f7f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.546998 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e (DEFAULT_TABLE table=default.Table_e605f1f30c8d4fdd8503b71227359798 [id=285208f892d84eea88ede0f96db5e7cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.547375 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a692f00ba244a80b9a6d07029f8d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.549261 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_da45cfc017de463a9a4b04db4cf3f710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.549674 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da45cfc017de463a9a4b04db4cf3f710 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.550882 446 master_service.cc:948] ReplaceTablet: received request to replace tablet b939a8568ac14c339258af925fcb809b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.551175 884 tablet_service.cc:1468] Processing CreateTablet for tablet ea00b4559e03410885180286b5e9fd10 (DEFAULT_TABLE table=default.Table_5d35d0525f924ed692497dbaf2287e8e [id=e7ad9b02d0f14bdfa3914844146a1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.551633 888 tablet_service.cc:1468] Processing CreateTablet for tablet adbb648c49e24b8085eacbe6e0a86991 (DEFAULT_TABLE table=default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.551936 891 tablet_service.cc:1468] Processing CreateTablet for tablet 798516e55ca747fcb2942026adb78af0 (DEFAULT_TABLE table=default.Table_60c45c3633104e8aa039d5d92633c213 [id=fd0fa3108c2b4816b502364824e4cc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.552110 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5860f7100a78476b8fa1d34f6c0b8d77 (DEFAULT_TABLE table=default.Table_4e8a8a6b9cde425c8326fd3609daf30f [id=606dee2396e645cd829e21ff5b18e546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.552415 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5860f7100a78476b8fa1d34f6c0b8d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.553165 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798516e55ca747fcb2942026adb78af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.556730 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adbb648c49e24b8085eacbe6e0a86991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.558462 1154 tablet_bootstrap.cc:492] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.558604 1154 ts_tablet_manager.cc:1397] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.001s
I20250629 01:51:42.559288 1154 raft_consensus.cc:357] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.559437 1154 raft_consensus.cc:383] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.559497 1154 raft_consensus.cc:738] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.559726 1154 consensus_queue.cc:260] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.560149 1154 ts_tablet_manager.cc:1428] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:42.560984 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea00b4559e03410885180286b5e9fd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.561690 1154 tablet_bootstrap.cc:492] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.565121 1154 tablet_bootstrap.cc:654] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:42.568015 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df66984acfa44a99bf69096aff6d49ab consensus state has no leader
I20250629 01:51:42.569149 1241 raft_consensus.cc:491] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.569340 1241 raft_consensus.cc:513] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.570057 1241 leader_election.cc:290] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.570650 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78a5f38231a34efaab177a23cf693d51" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:42.571033 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 431dc961cdcf4cd38b521d8d8fc6f406 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:42.571094 712 leader_election.cc:343] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.571274 1156 tablet_replica.cc:331] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.571458 1156 raft_consensus.cc:2241] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.571573 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a5f38231a34efaab177a23cf693d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.571619 1156 raft_consensus.cc:2270] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.571280 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 4562bba09498449fbff9b7c60bad58fc with delete_type TABLET_DATA_DELETED (replaced by tablet 97e03c52aeb4432cab002abdae5109ba) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.572080 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5b835ff8624ebf8956eeb965ce5950) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.572523 754 tablet_service.cc:1515] Processing DeleteTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 with delete_type TABLET_DATA_DELETED (replaced by tablet 36fcb819c61c4fabbb0da295d0002998) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.572655 1156 ts_tablet_manager.cc:1905] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.573144 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e03c52aeb4432cab002abdae5109ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.573562 754 tablet_service.cc:1515] Processing DeleteTablet for tablet d1956051272e450ab2ba3322ba23ab69 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d072bf72044dc48d8090660ecb7d47) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.573697 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 829c64e2c71347f4af2c8870cdd8af31
I20250629 01:51:42.574838 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.576637 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.582016 1156 ts_tablet_manager.cc:1918] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:42.582264 710 leader_election.cc:343] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 78a5f38231a34efaab177a23cf693d51
I20250629 01:51:42.582302 1406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:42.582394 710 leader_election.cc:304] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:42.582640 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c (DEFAULT_TABLE table=default.Table_adf0dc244a084dcf82f6cb6f22ecc128 [id=d3d515038aa84669a3a64824de88f06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.582732 1241 raft_consensus.cc:2747] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:42.582975 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf553b291854a6f9e7ed78b5db5f18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.583024 1156 log.cc:1199] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/431dc961cdcf4cd38b521d8d8fc6f406
I20250629 01:51:42.583725 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b939a8568ac14c339258af925fcb809b
I20250629 01:51:42.584072 446 catalog_manager.cc:6652] ReplaceTablet: tablet b939a8568ac14c339258af925fcb809b deleted and replaced by tablet 34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:42.584322 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3618e21295a84926bda02aa1d8e90a32 (DEFAULT_TABLE table=default.Table_7e0ccdf1ddb349558670c1d0a61d5cf2 [id=15f7a887c8224335aaf55ad0469bbf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.584373 1277 fs_manager.cc:730] Time spent opening block manager: real 0.077s user 0.002s sys 0.002s
I20250629 01:51:42.584631 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618e21295a84926bda02aa1d8e90a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.585043 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0019f37bf9d34f969b4ef89adc5dfb9d" } new_table_name: "default.Table_6b6adcddb17145a7a5a51e2f83ada474" modify_external_catalogs: true
I20250629 01:51:42.584527 1277 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
uuid: "f304077528b34c61857a659e5873631d"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:42.586259 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_92068d6a585a434da503666ff8ac9536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:42.586519 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_04cc9d29a25645f0b3facf31bd7a9f57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.586727 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92068d6a585a434da503666ff8ac9536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:42.586936 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04cc9d29a25645f0b3facf31bd7a9f57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.585460 1277 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:42.588033 882 tablet_service.cc:1515] Processing DeleteTablet for tablet b939a8568ac14c339258af925fcb809b with delete_type TABLET_DATA_DELETED (replaced by tablet 34172d05ae40417494cd8f67f4cb981e) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.583321 1156 ts_tablet_manager.cc:1939] T 431dc961cdcf4cd38b521d8d8fc6f406 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.588968 1156 tablet_replica.cc:331] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.589166 1156 raft_consensus.cc:2241] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.589358 1156 raft_consensus.cc:2270] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.589622 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e03c52aeb4432cab002abdae5109ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.589901 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet f7bb791d4a38481aa05a4dbbe3577461 with delete_type TABLET_DATA_DELETED (replaced by tablet 36fcb819c61c4fabbb0da295d0002998) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.590122 1158 tablet_replica.cc:331] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.590323 1158 raft_consensus.cc:2241] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.590555 1158 raft_consensus.cc:2270] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.591650 1158 ts_tablet_manager.cc:1905] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.593937 1157 ts_tablet_manager.cc:1918] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.594054 1157 log.cc:1199] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f7bb791d4a38481aa05a4dbbe3577461
I20250629 01:51:42.594302 886 tablet_service.cc:1468] Processing CreateTablet for tablet df66984acfa44a99bf69096aff6d49ab (DEFAULT_TABLE table=default.Table_abc4bc33793d4bee824f97f24a172593 [id=8a6a13c4f09547aa8ca713ab9892f557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.594393 1157 ts_tablet_manager.cc:1939] T f7bb791d4a38481aa05a4dbbe3577461 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:42.594641 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df66984acfa44a99bf69096aff6d49ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.595079 1159 tablet_bootstrap.cc:492] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.595291 1159 ts_tablet_manager.cc:1397] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.138s user 0.005s sys 0.002s
I20250629 01:51:42.596086 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f7bb791d4a38481aa05a4dbbe3577461 (table default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]) successfully deleted
W20250629 01:51:42.598362 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 829c64e2c71347f4af2c8870cdd8af31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829c64e2c71347f4af2c8870cdd8af31 already in progress: creating tablet
I20250629 01:51:42.596102 1159 raft_consensus.cc:357] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.599030 1159 raft_consensus.cc:383] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.599124 1159 raft_consensus.cc:738] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.599689 1156 ts_tablet_manager.cc:1905] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.600278 1159 consensus_queue.cc:260] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.602830 1159 ts_tablet_manager.cc:1428] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250629 01:51:42.603375 1159 tablet_bootstrap.cc:492] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:42.605392 1159 tablet_bootstrap.cc:654] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.606173 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:32789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111),
I20250629 01:51:42.606344 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250629 01:51:42.606508 754 tablet_service.cc:1515] Processing DeleteTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.607108 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 (DEFAULT_TABLE table=default.Table_f641040c5d3f4fabb4642ae6c32b10c7 [id=7607f62687244e718732520ce1342b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.607488 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7c1f1af39d47c191d34dc026f7f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.608016 1158 ts_tablet_manager.cc:1918] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.610821 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f248b655abe641dbbf9eb656ff017e53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.611377 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f248b655abe641dbbf9eb656ff017e53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.611465 1158 log.cc:1199] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/97e03c52aeb4432cab002abdae5109ba
I20250629 01:51:42.611969 1158 ts_tablet_manager.cc:1939] T 97e03c52aeb4432cab002abdae5109ba P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.612433 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d1f1005233b74726b856dba70c0a4b8b" } new_table_name: "default.Table_328919206d1e44838bc19f8a2ff9d371" modify_external_catalogs: true
I20250629 01:51:42.612901 1156 ts_tablet_manager.cc:1918] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.613010 1156 log.cc:1199] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/78a5f38231a34efaab177a23cf693d51
I20250629 01:51:42.613324 1156 ts_tablet_manager.cc:1939] T 78a5f38231a34efaab177a23cf693d51 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.613621 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_821e341e379740a59615270511038004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:42.613862 1154 tablet_bootstrap.cc:492] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.614008 1154 ts_tablet_manager.cc:1397] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.003s
W20250629 01:51:42.614012 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_821e341e379740a59615270511038004 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.614298 1156 tablet_replica.cc:331] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.614490 1156 raft_consensus.cc:2241] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.614643 1156 raft_consensus.cc:2270] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.614856 1154 raft_consensus.cc:357] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.615051 1154 raft_consensus.cc:383] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.615163 1154 raft_consensus.cc:738] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.615415 1154 consensus_queue.cc:260] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.616310 1154 ts_tablet_manager.cc:1428] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:42.616678 1154 tablet_bootstrap.cc:492] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.618616 1154 tablet_bootstrap.cc:654] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.619143 1158 tablet_replica.cc:331] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.620384 1158 raft_consensus.cc:2241] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.620582 1158 raft_consensus.cc:2270] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.621238 1158 ts_tablet_manager.cc:1905] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.623116 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.623404 1156 ts_tablet_manager.cc:1905] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.624608 891 tablet_service.cc:1468] Processing CreateTablet for tablet 81b32bac5a2144d1943285f5f446d53d (DEFAULT_TABLE table=default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.624984 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b32bac5a2144d1943285f5f446d53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.625870 455 master_service.cc:948] ReplaceTablet: received request to replace tablet b939a8568ac14c339258af925fcb809b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.629560 1158 ts_tablet_manager.cc:1918] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.632220 1156 ts_tablet_manager.cc:1918] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.632817 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c28ea445cec4b35be800dbc8c37af7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.634564 1156 log.cc:1199] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/7ae96f2da36744198e3d38cb07bb47a0
I20250629 01:51:42.635084 1156 ts_tablet_manager.cc:1939] T 7ae96f2da36744198e3d38cb07bb47a0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.633474 1154 tablet_bootstrap.cc:492] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.636582 1154 ts_tablet_manager.cc:1397] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250629 01:51:42.637063 1156 tablet_replica.cc:331] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.637140 1154 raft_consensus.cc:357] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.637305 1154 raft_consensus.cc:383] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.637362 1154 raft_consensus.cc:738] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.637576 1156 raft_consensus.cc:2241] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.637563 1154 consensus_queue.cc:260] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.637943 1154 ts_tablet_manager.cc:1428] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:42.637969 1156 raft_consensus.cc:2270] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.638541 1154 tablet_bootstrap.cc:492] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:42.639988 1154 tablet_bootstrap.cc:654] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.641171 1158 log.cc:1199] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f7bb791d4a38481aa05a4dbbe3577461
I20250629 01:51:42.641520 1158 ts_tablet_manager.cc:1939] T f7bb791d4a38481aa05a4dbbe3577461 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:42.642719 746 tablet_service.cc:1468] Processing CreateTablet for tablet 798516e55ca747fcb2942026adb78af0 (DEFAULT_TABLE table=default.Table_60c45c3633104e8aa039d5d92633c213 [id=fd0fa3108c2b4816b502364824e4cc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.643049 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798516e55ca747fcb2942026adb78af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.643029 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e6b1ecf098ab4e498391e79cfb1c60b0 (table default.Table_7b25659fe37948109f17e9e6b15518cc [id=9a2d24af85fb40b3a4c49f37cca3ab62]) successfully deleted
W20250629 01:51:42.643244 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet dd6013bb40c248f29f40eab2c3008f6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd6013bb40c248f29f40eab2c3008f6b already in progress: creating tablet
I20250629 01:51:42.643296 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e (DEFAULT_TABLE table=default.Table_e605f1f30c8d4fdd8503b71227359798 [id=285208f892d84eea88ede0f96db5e7cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.643338 1364 raft_consensus.cc:491] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:42.643416 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d61687e19d994e87b09e55bd3b9476d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61687e19d994e87b09e55bd3b9476d7 already in progress: creating tablet
I20250629 01:51:42.643616 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a692f00ba244a80b9a6d07029f8d27e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:42.643604 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 69980d5bf37d4001a6196b22c34ccc94 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69980d5bf37d4001a6196b22c34ccc94 already in progress: creating tablet
I20250629 01:51:42.643471 1364 raft_consensus.cc:513] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:42.643985 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 69f0ae4c51924370896c624affde6761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f0ae4c51924370896c624affde6761 already in progress: creating tablet
I20250629 01:51:42.644181 1364 leader_election.cc:290] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:42.644326 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 798516e55ca747fcb2942026adb78af0 (DEFAULT_TABLE table=default.Table_60c45c3633104e8aa039d5d92633c213 [id=fd0fa3108c2b4816b502364824e4cc12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.644594 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 798516e55ca747fcb2942026adb78af0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.644793 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28ea445cec4b35be800dbc8c37af7c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:42.644984 916 raft_consensus.cc:2466] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:42.645298 988 leader_election.cc:304] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:42.646634 1364 raft_consensus.cc:2802] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:42.646732 1364 raft_consensus.cc:491] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:42.646816 1364 raft_consensus.cc:3058] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.648327 1021 tablet_service.cc:1468] Processing CreateTablet for tablet ea00b4559e03410885180286b5e9fd10 (DEFAULT_TABLE table=default.Table_5d35d0525f924ed692497dbaf2287e8e [id=e7ad9b02d0f14bdfa3914844146a1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.648350 743 tablet_service.cc:1468] Processing CreateTablet for tablet ea00b4559e03410885180286b5e9fd10 (DEFAULT_TABLE table=default.Table_5d35d0525f924ed692497dbaf2287e8e [id=e7ad9b02d0f14bdfa3914844146a1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.648598 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea00b4559e03410885180286b5e9fd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.648638 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea00b4559e03410885180286b5e9fd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.649289 1364 raft_consensus.cc:513] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.650209 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c28ea445cec4b35be800dbc8c37af7c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:42.650887 916 raft_consensus.cc:3058] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:42.652290 1032 tablet_service.cc:1468] Processing CreateTablet for tablet adbb648c49e24b8085eacbe6e0a86991 (DEFAULT_TABLE table=default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.652326 757 tablet_service.cc:1468] Processing CreateTablet for tablet adbb648c49e24b8085eacbe6e0a86991 (DEFAULT_TABLE table=default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.652550 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adbb648c49e24b8085eacbe6e0a86991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.652631 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adbb648c49e24b8085eacbe6e0a86991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.653220 916 raft_consensus.cc:2466] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:42.653518 988 leader_election.cc:304] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:42.653800 1311 raft_consensus.cc:2802] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:42.653869 1156 ts_tablet_manager.cc:1905] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.653915 1311 raft_consensus.cc:695] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:42.654163 1311 consensus_queue.cc:237] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.654776 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet d1956051272e450ab2ba3322ba23ab69 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d072bf72044dc48d8090660ecb7d47) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.655047 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 5860f7100a78476b8fa1d34f6c0b8d77 (DEFAULT_TABLE table=default.Table_4e8a8a6b9cde425c8326fd3609daf30f [id=606dee2396e645cd829e21ff5b18e546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.655336 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5860f7100a78476b8fa1d34f6c0b8d77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.655423 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 (DEFAULT_TABLE table=default.Table_f641040c5d3f4fabb4642ae6c32b10c7 [id=7607f62687244e718732520ce1342b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.655763 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7c1f1af39d47c191d34dc026f7f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.656332 744 tablet_service.cc:1468] Processing CreateTablet for tablet 5860f7100a78476b8fa1d34f6c0b8d77 (DEFAULT_TABLE table=default.Table_4e8a8a6b9cde425c8326fd3609daf30f [id=606dee2396e645cd829e21ff5b18e546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.656656 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5860f7100a78476b8fa1d34f6c0b8d77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:42.657732 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f3bf5fd9b674eeb8a5a029152cbae13 already in progress: creating tablet
I20250629 01:51:42.657900 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 97e03c52aeb4432cab002abdae5109ba (table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]) successfully deleted
W20250629 01:51:42.658187 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet dd6013bb40c248f29f40eab2c3008f6b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd6013bb40c248f29f40eab2c3008f6b already in progress: creating tablet
I20250629 01:51:42.658986 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 431dc961cdcf4cd38b521d8d8fc6f406 (table default.Table_a37f4562238b452abf51e6e3d547ca36 [id=c6a48bb9a6874c40b2f7679f2db9e756]) successfully deleted
I20250629 01:51:42.659142 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 78a5f38231a34efaab177a23cf693d51 (table default.Table_601ff9b6b68b4acb9206a64afa4339a8 [id=486534f71b544cee8fd56472d6806f80]) successfully deleted
W20250629 01:51:42.659289 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4562bba09498449fbff9b7c60bad58fc because the tablet was not found. No further retry: Not found: Tablet not found: 4562bba09498449fbff9b7c60bad58fc
I20250629 01:51:42.659516 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 7ae96f2da36744198e3d38cb07bb47a0 (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:42.660384 1157 tablet_replica.cc:331] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.660564 1157 raft_consensus.cc:2241] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.660605 1241 raft_consensus.cc:491] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.660704 1157 raft_consensus.cc:2270] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.660755 1241 raft_consensus.cc:513] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.661000 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.661257 1241 leader_election.cc:290] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.661319 755 tablet_service.cc:1468] Processing CreateTablet for tablet 81b32bac5a2144d1943285f5f446d53d (DEFAULT_TABLE table=default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.661610 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b32bac5a2144d1943285f5f446d53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.663427 1159 tablet_bootstrap.cc:492] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.663539 1159 ts_tablet_manager.cc:1397] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.000s
I20250629 01:51:42.663580 756 tablet_service.cc:1468] Processing CreateTablet for tablet df66984acfa44a99bf69096aff6d49ab (DEFAULT_TABLE table=default.Table_abc4bc33793d4bee824f97f24a172593 [id=8a6a13c4f09547aa8ca713ab9892f557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.663877 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df66984acfa44a99bf69096aff6d49ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.664243 1159 raft_consensus.cc:357] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.664335 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97e03c52aeb4432cab002abdae5109ba" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:42.664335 750 tablet_service.cc:1468] Processing CreateTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c (DEFAULT_TABLE table=default.Table_adf0dc244a084dcf82f6cb6f22ecc128 [id=d3d515038aa84669a3a64824de88f06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.664404 1159 raft_consensus.cc:383] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.664477 1159 raft_consensus.cc:738] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.664791 1159 consensus_queue.cc:260] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.665366 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf553b291854a6f9e7ed78b5db5f18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.665580 1159 ts_tablet_manager.cc:1428] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:42.666898 1157 ts_tablet_manager.cc:1905] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:42.667356 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b939a8568ac14c339258af925fcb809b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b939a8568ac14c339258af925fcb809b already in progress: creating tablet
I20250629 01:51:42.667711 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 81b32bac5a2144d1943285f5f446d53d (DEFAULT_TABLE table=default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.668653 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3618e21295a84926bda02aa1d8e90a32 (DEFAULT_TABLE table=default.Table_7e0ccdf1ddb349558670c1d0a61d5cf2 [id=15f7a887c8224335aaf55ad0469bbf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.668942 747 tablet_service.cc:1515] Processing DeleteTablet for tablet b939a8568ac14c339258af925fcb809b with delete_type TABLET_DATA_DELETED (replaced by tablet 34172d05ae40417494cd8f67f4cb981e) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.669080 1159 tablet_bootstrap.cc:492] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:42.669193 712 leader_election.cc:343] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.670486 1159 tablet_bootstrap.cc:654] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.672041 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0" } modify_external_catalogs: true
I20250629 01:51:42.672222 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0" } modify_external_catalogs: true
I20250629 01:51:42.673657 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4559cd80e6874b7fb3e1ea565b90fd03" } modify_external_catalogs: true
I20250629 01:51:42.673799 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4559cd80e6874b7fb3e1ea565b90fd03" } modify_external_catalogs: true
I20250629 01:51:42.674202 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fa24ba8423a34245b118894025a8b4f4" } modify_external_catalogs: true
I20250629 01:51:42.674332 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fa24ba8423a34245b118894025a8b4f4" } modify_external_catalogs: true
I20250629 01:51:42.676373 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d219b9a9062745bca33e3a0fcf553be0" } modify_external_catalogs: true
I20250629 01:51:42.676563 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d219b9a9062745bca33e3a0fcf553be0" } modify_external_catalogs: true
I20250629 01:51:42.678138 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bf61fc85ef4b405180f3b4621489590e" } new_table_name: "default.Table_cf2dc25aa17a42fb91e6903713ca0d6c" modify_external_catalogs: true
I20250629 01:51:42.679461 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e (DEFAULT_TABLE table=default.Table_e605f1f30c8d4fdd8503b71227359798 [id=285208f892d84eea88ede0f96db5e7cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.681135 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618e21295a84926bda02aa1d8e90a32. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:42.685572 986 leader_election.cc:343] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0c28ea445cec4b35be800dbc8c37af7c
I20250629 01:51:42.686210 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a692f00ba244a80b9a6d07029f8d27e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.687474 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f7bb791d4a38481aa05a4dbbe3577461 (table default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]) successfully deleted
I20250629 01:51:42.687500 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f36b714b857b4ffdb47fc4eb3261e7a1" } new_table_name: "default.Table_386bd2d6e963443cb19c211c4e6e0120" modify_external_catalogs: true
I20250629 01:51:42.688565 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81b32bac5a2144d1943285f5f446d53d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.689162 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3618e21295a84926bda02aa1d8e90a32 (DEFAULT_TABLE table=default.Table_7e0ccdf1ddb349558670c1d0a61d5cf2 [id=15f7a887c8224335aaf55ad0469bbf12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.689424 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:42.690577 710 leader_election.cc:343] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 97e03c52aeb4432cab002abdae5109ba
I20250629 01:51:42.690708 710 leader_election.cc:304] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:42.691002 1263 raft_consensus.cc:2747] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:42.694600 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9b3067ec3f994a44bf4ecacbc937f414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.695060 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b3067ec3f994a44bf4ecacbc937f414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.697314 1157 ts_tablet_manager.cc:1918] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.697316 1159 tablet_bootstrap.cc:492] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.698101 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5b835ff8624ebf8956eeb965ce5950) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.698338 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3618e21295a84926bda02aa1d8e90a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.698462 1155 tablet_bootstrap.cc:492] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.698143 1159 ts_tablet_manager.cc:1397] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.007s
I20250629 01:51:42.698599 1155 ts_tablet_manager.cc:1397] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.162s user 0.008s sys 0.020s
I20250629 01:51:42.699312 1155 raft_consensus.cc:357] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.699510 1155 raft_consensus.cc:383] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.699586 1155 raft_consensus.cc:738] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.699309 1382 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:42.699803 1155 consensus_queue.cc:260] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.700021 1339 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:42.700227 1155 ts_tablet_manager.cc:1428] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:42.702519 1155 tablet_bootstrap.cc:492] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.705587 1364 leader_election.cc:290] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:42.707641 1156 ts_tablet_manager.cc:1918] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.707744 1156 log.cc:1199] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f7bb791d4a38481aa05a4dbbe3577461
I20250629 01:51:42.708034 1156 ts_tablet_manager.cc:1939] T f7bb791d4a38481aa05a4dbbe3577461 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.708815 1156 tablet_replica.cc:331] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.708997 1156 raft_consensus.cc:2241] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.709131 1156 raft_consensus.cc:2270] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.709267 1155 tablet_bootstrap.cc:654] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.709594 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet b939a8568ac14c339258af925fcb809b with delete_type TABLET_DATA_DELETED (replaced by tablet 34172d05ae40417494cd8f67f4cb981e) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:42.709868 986 leader_election.cc:343] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0c28ea445cec4b35be800dbc8c37af7c
I20250629 01:51:42.710276 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 7ae96f2da36744198e3d38cb07bb47a0 with delete_type TABLET_DATA_DELETED (replaced by tablet bd5b835ff8624ebf8956eeb965ce5950) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.710402 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f7bb791d4a38481aa05a4dbbe3577461 (table default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]) successfully deleted
I20250629 01:51:42.710508 1025 tablet_service.cc:1468] Processing CreateTablet for tablet df66984acfa44a99bf69096aff6d49ab (DEFAULT_TABLE table=default.Table_abc4bc33793d4bee824f97f24a172593 [id=8a6a13c4f09547aa8ca713ab9892f557]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.711038 1156 ts_tablet_manager.cc:1905] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.712715 449 master_service.cc:948] ReplaceTablet: received request to replace tablet e6b1ecf098ab4e498391e79cfb1c60b0 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.712878 1157 log.cc:1199] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0c28ea445cec4b35be800dbc8c37af7c
I20250629 01:51:42.713325 1157 ts_tablet_manager.cc:1939] T 0c28ea445cec4b35be800dbc8c37af7c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:42.713681 1159 raft_consensus.cc:357] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.714668 1159 raft_consensus.cc:383] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.714900 1159 raft_consensus.cc:738] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.715400 1159 consensus_queue.cc:260] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.715185 1157 tablet_replica.cc:331] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.716058 1157 raft_consensus.cc:2241] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.717348 1157 raft_consensus.cc:2270] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.718403 1157 ts_tablet_manager.cc:1905] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.717746 1159 ts_tablet_manager.cc:1428] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250629 01:51:42.719583 1159 tablet_bootstrap.cc:492] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:42.716076 1156 ts_tablet_manager.cc:1918] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.721330 1156 log.cc:1199] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/97e03c52aeb4432cab002abdae5109ba
I20250629 01:51:42.721635 1156 ts_tablet_manager.cc:1939] T 97e03c52aeb4432cab002abdae5109ba P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.722362 1156 tablet_replica.cc:331] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.722532 1156 raft_consensus.cc:2241] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.722690 1156 raft_consensus.cc:2270] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.724906 1156 ts_tablet_manager.cc:1905] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.726904 884 tablet_service.cc:1468] Processing CreateTablet for tablet 432680cae61946af9859760fe23110ec (DEFAULT_TABLE table=default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 [id=2dfa777b5eab4acb9adfd35ab56c17d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.727432 894 tablet_service.cc:1468] Processing CreateTablet for tablet 541728ff8dc3471ba90784b4a769c8e1 (DEFAULT_TABLE table=default.Table_92068d6a585a434da503666ff8ac9536 [id=fd0ea1adec3e487ab647680c75b7fde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.727576 888 tablet_service.cc:1468] Processing CreateTablet for tablet 47d4287317ec4788bd96ad174c31748d (DEFAULT_TABLE table=default.Table_58a60b60dab64affa9d8b5b1acf53326 [id=d524cd9f7aba4981b939a770099325c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.727799 886 tablet_service.cc:1468] Processing CreateTablet for tablet 943f584fb9554fa4b65e5302443cec92 (DEFAULT_TABLE table=default.Table_da45cfc017de463a9a4b04db4cf3f710 [id=4bfe4f0aff75478fbd422f8c1f46d45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.728057 895 tablet_service.cc:1468] Processing CreateTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 (DEFAULT_TABLE table=default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.728319 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2d44151acb0d4bf4a630c086b1ac8ff7
I20250629 01:51:42.728077 891 tablet_service.cc:1468] Processing CreateTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 (DEFAULT_TABLE table=default.Table_04cc9d29a25645f0b3facf31bd7a9f57 [id=395f2b984ff9443a90e78e8d44bca991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.728528 893 tablet_service.cc:1468] Processing CreateTablet for tablet 34172d05ae40417494cd8f67f4cb981e (DEFAULT_TABLE table=default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.728652 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432680cae61946af9859760fe23110ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.728873 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d4287317ec4788bd96ad174c31748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.729053 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34172d05ae40417494cd8f67f4cb981e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.729001 882 tablet_service.cc:1468] Processing CreateTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 (DEFAULT_TABLE table=default.Table_f128352ab4bd4f258d90d9a4022577a8 [id=b516da4148384818bcc69e4259f20ef9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:42.729394 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f consensus state has no leader
W20250629 01:51:42.729831 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1fe3880e51345269e852014d0d60bb6 consensus state has no leader
I20250629 01:51:42.731995 885 tablet_service.cc:1468] Processing CreateTablet for tablet e962a020664b42ea918e5952dc43aade (DEFAULT_TABLE table=default.Table_e0533c0c40ca4af1be6ac30ee37f3aed [id=1d5fd1e402a54d31b3fea01d38db6918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.732429 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0c28ea445cec4b35be800dbc8c37af7c (table default.Table_7a50e6b289ed401092480b4d315d2840 [id=c1e051380c514aa59fd3d4d84ee465d4]) successfully deleted
I20250629 01:51:42.734445 754 tablet_service.cc:1468] Processing CreateTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 (DEFAULT_TABLE table=default.Table_04cc9d29a25645f0b3facf31bd7a9f57 [id=395f2b984ff9443a90e78e8d44bca991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.734853 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 683c87ea2ad44cecb2366d4a846d9195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.735908 750 tablet_service.cc:1468] Processing CreateTablet for tablet e962a020664b42ea918e5952dc43aade (DEFAULT_TABLE table=default.Table_e0533c0c40ca4af1be6ac30ee37f3aed [id=1d5fd1e402a54d31b3fea01d38db6918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.736129 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 97e03c52aeb4432cab002abdae5109ba (table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]) successfully deleted
I20250629 01:51:42.736245 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e962a020664b42ea918e5952dc43aade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.736425 757 tablet_service.cc:1468] Processing CreateTablet for tablet 943f584fb9554fa4b65e5302443cec92 (DEFAULT_TABLE table=default.Table_da45cfc017de463a9a4b04db4cf3f710 [id=4bfe4f0aff75478fbd422f8c1f46d45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.736811 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943f584fb9554fa4b65e5302443cec92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.738423 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3d334b0f295e42378d74e5e746d911e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.739177 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d334b0f295e42378d74e5e746d911e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.739235 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d9c78256954c4e8191c1373909f6b086"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.739621 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9c78256954c4e8191c1373909f6b086 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.740288 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942f7bbe29f94083b2494aab8fbcc0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.740496 756 tablet_service.cc:1468] Processing CreateTablet for tablet 432680cae61946af9859760fe23110ec (DEFAULT_TABLE table=default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 [id=2dfa777b5eab4acb9adfd35ab56c17d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.740801 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432680cae61946af9859760fe23110ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.740996 889 tablet_service.cc:1468] Processing CreateTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 (DEFAULT_TABLE table=default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.741235 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943f584fb9554fa4b65e5302443cec92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.742378 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541728ff8dc3471ba90784b4a769c8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.743255 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d925ffce89b40258b29ec021bfa5f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.743954 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e962a020664b42ea918e5952dc43aade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.744140 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 (DEFAULT_TABLE table=default.Table_f128352ab4bd4f258d90d9a4022577a8 [id=b516da4148384818bcc69e4259f20ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.744475 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d925ffce89b40258b29ec021bfa5f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.744567 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a485e0e6e2f04aa6bd67a62f6d30be26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.745589 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 683c87ea2ad44cecb2366d4a846d9195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.747218 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.747864 743 tablet_service.cc:1468] Processing CreateTablet for tablet 47d4287317ec4788bd96ad174c31748d (DEFAULT_TABLE table=default.Table_58a60b60dab64affa9d8b5b1acf53326 [id=d524cd9f7aba4981b939a770099325c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.747995 887 tablet_service.cc:1468] Processing CreateTablet for tablet 35990d206ba64159ab05ab7b7b19d064 (DEFAULT_TABLE table=default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 [id=8d49193b68504e77818e373ecad346b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.748342 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35990d206ba64159ab05ab7b7b19d064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.748355 744 tablet_service.cc:1468] Processing CreateTablet for tablet 34172d05ae40417494cd8f67f4cb981e (DEFAULT_TABLE table=default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.748852 747 tablet_service.cc:1468] Processing CreateTablet for tablet 541728ff8dc3471ba90784b4a769c8e1 (DEFAULT_TABLE table=default.Table_92068d6a585a434da503666ff8ac9536 [id=fd0ea1adec3e487ab647680c75b7fde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.749269 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.749600 752 tablet_service.cc:1468] Processing CreateTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 (DEFAULT_TABLE table=default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.750399 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.750707 741 tablet_service.cc:1468] Processing CreateTablet for tablet 35990d206ba64159ab05ab7b7b19d064 (DEFAULT_TABLE table=default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 [id=8d49193b68504e77818e373ecad346b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.751708 755 tablet_service.cc:1468] Processing CreateTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 (DEFAULT_TABLE table=default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.752190 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d4287317ec4788bd96ad174c31748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.754880 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35990d206ba64159ab05ab7b7b19d064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.756232 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942f7bbe29f94083b2494aab8fbcc0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.759487 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a485e0e6e2f04aa6bd67a62f6d30be26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.760228 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34172d05ae40417494cd8f67f4cb981e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.761631 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:42.762219 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b7c7f2ac44734a39ab6434a74c91b9e4
I20250629 01:51:42.762797 746 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.763037 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541728ff8dc3471ba90784b4a769c8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.764268 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:42.766160 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 81b32bac5a2144d1943285f5f446d53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.766476 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 34172d05ae40417494cd8f67f4cb981e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.768771 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9b82f19de1c74fcd9973b4a4084712e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.769213 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b82f19de1c74fcd9973b4a4084712e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.769361 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ed3568508cfb4c4b9297f986d7a9963e" } new_table_name: "default.Table_7ac85c469b3c4de2aa2320e2261b0dcd" modify_external_catalogs: true
I20250629 01:51:42.770287 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a47465ab39a14717b174e77b563b4100"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.770668 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a47465ab39a14717b174e77b563b4100 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.776925 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6b7f597bfe3947f884cf5e129049c99b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.777765 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b7f597bfe3947f884cf5e129049c99b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.779837 1277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:42.780846 1277 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:42.781100 1277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:42.781934 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 34172d05ae40417494cd8f67f4cb981e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.784173 883 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.784880 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 81b32bac5a2144d1943285f5f446d53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.789594 1263 raft_consensus.cc:491] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.789733 1263 raft_consensus.cc:513] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.790257 1263 leader_election.cc:290] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:42.791177 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64477468c6da4b4ba282d8530eb5cbb5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:42.791661 712 leader_election.cc:343] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:42.798938 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b7a640451934bff8672cd88dc86c529 consensus state has no leader
I20250629 01:51:42.802811 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_6b6adcddb17145a7a5a51e2f83ada474 [id=60094a0a7c134476a2a4e44d8555c5d6] alter complete (version 3)
I20250629 01:51:42.806638 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1b80dba0f8d04043b95a7aaa176636b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.807070 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b80dba0f8d04043b95a7aaa176636b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.808077 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aa33973948ab4d68b062deca5d60a668" } new_table_name: "default.Table_de5d37756c7649fc9bf3ae6d5adf922f" modify_external_catalogs: true
I20250629 01:51:42.809109 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7b31a72aec3c4710971cd48931f10395"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.809489 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b31a72aec3c4710971cd48931f10395 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.810708 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f0a26d9ea3e8407da5ca0813db26beb3" } modify_external_catalogs: true
I20250629 01:51:42.810846 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f0a26d9ea3e8407da5ca0813db26beb3" } modify_external_catalogs: true
I20250629 01:51:42.812351 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 351f4a7ae7e244999dda81d282de505d from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.812541 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df66984acfa44a99bf69096aff6d49ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.807487 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c (DEFAULT_TABLE table=default.Table_adf0dc244a084dcf82f6cb6f22ecc128 [id=d3d515038aa84669a3a64824de88f06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.817616 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 97e03c52aeb4432cab002abdae5109ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.820708 1156 ts_tablet_manager.cc:1918] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.824179 1156 log.cc:1199] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d1956051272e450ab2ba3322ba23ab69
I20250629 01:51:42.824574 1156 ts_tablet_manager.cc:1939] T d1956051272e450ab2ba3322ba23ab69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.825462 1156 tablet_replica.cc:331] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.825428 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d1956051272e450ab2ba3322ba23ab69 (table default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]) successfully deleted
I20250629 01:51:42.825688 1156 raft_consensus.cc:2241] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.825881 1156 raft_consensus.cc:2270] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.827013 1156 ts_tablet_manager.cc:1905] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.828994 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 3ba0e267f48a4bf1b58fb2339694719f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.832856 1155 tablet_bootstrap.cc:492] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:42.832978 1155 ts_tablet_manager.cc:1397] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.131s user 0.012s sys 0.000s
W20250629 01:51:42.833571 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97439c6a6e844610b55b899966ae5b67 consensus state has no leader
I20250629 01:51:42.836136 757 tablet_service.cc:1468] Processing CreateTablet for tablet 013ab393a0f14bdc89389349bcc5d24b (DEFAULT_TABLE table=default.Table_f248b655abe641dbbf9eb656ff017e53 [id=55cb83afbc4f461f8946c50d83054bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.836478 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 013ab393a0f14bdc89389349bcc5d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.836589 1155 raft_consensus.cc:357] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.836747 1155 raft_consensus.cc:383] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.836831 1155 raft_consensus.cc:738] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.837019 1155 consensus_queue.cc:260] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.837483 1155 ts_tablet_manager.cc:1428] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:42.837740 1155 tablet_bootstrap.cc:492] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:42.838300 755 tablet_service.cc:1468] Processing CreateTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 (DEFAULT_TABLE table=default.Table_9b3067ec3f994a44bf4ecacbc937f414 [id=5d8dd5f1acfd4c57b7080fc663a1d32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.838618 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ce1bd4af434f7e99092aafc0ab0118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.840298 1159 tablet_bootstrap.cc:654] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.841038 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c712d978a3848b9bb3abdbce8314dec from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.841436 754 tablet_service.cc:1468] Processing CreateTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 (DEFAULT_TABLE table=default.Table_821e341e379740a59615270511038004 [id=910c490dcc294f58b93e5b6c4846d263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.841697 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196f1bd233ab4399a9d22a55b8bc2635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.841842 1387 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:42.842126 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 97439c6a6e844610b55b899966ae5b67) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.816780 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 (DEFAULT_TABLE table=default.Table_04cc9d29a25645f0b3facf31bd7a9f57 [id=395f2b984ff9443a90e78e8d44bca991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.844918 1158 tablet_replica.cc:331] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:42.844946 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 47d4287317ec4788bd96ad174c31748d (DEFAULT_TABLE table=default.Table_58a60b60dab64affa9d8b5b1acf53326 [id=d524cd9f7aba4981b939a770099325c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.845182 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 (DEFAULT_TABLE table=default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.845310 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 (DEFAULT_TABLE table=default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.844950 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d44151acb0d4bf4a630c086b1ac8ff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.846314 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6991a3825f4547818c6b688dfc8ff59c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.846756 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6991a3825f4547818c6b688dfc8ff59c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.849589 1155 tablet_bootstrap.cc:654] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:42.851342 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0b7a640451934bff8672cd88dc86c529 (table default.Table_7ac85c469b3c4de2aa2320e2261b0dcd [id=e735a7d5da48458da86326bf01640f70], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0b7a640451934bff8672cd88dc86c529 consensus state has no leader
I20250629 01:51:42.852166 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf553b291854a6f9e7ed78b5db5f18c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.852946 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 943f584fb9554fa4b65e5302443cec92 (DEFAULT_TABLE table=default.Table_da45cfc017de463a9a4b04db4cf3f710 [id=4bfe4f0aff75478fbd422f8c1f46d45e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.853077 1018 tablet_service.cc:1468] Processing CreateTablet for tablet e962a020664b42ea918e5952dc43aade (DEFAULT_TABLE table=default.Table_e0533c0c40ca4af1be6ac30ee37f3aed [id=1d5fd1e402a54d31b3fea01d38db6918]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.858493 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:42.836129 1156 ts_tablet_manager.cc:1918] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.858949 1156 log.cc:1199] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e6b1ecf098ab4e498391e79cfb1c60b0
I20250629 01:51:42.859915 745 tablet_service.cc:1515] Processing DeleteTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 97439c6a6e844610b55b899966ae5b67) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.860476 1156 ts_tablet_manager.cc:1939] T e6b1ecf098ab4e498391e79cfb1c60b0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.861625 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e6b1ecf098ab4e498391e79cfb1c60b0 (table default.Table_7b25659fe37948109f17e9e6b15518cc [id=9a2d24af85fb40b3a4c49f37cca3ab62]) successfully deleted
I20250629 01:51:42.864674 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 34172d05ae40417494cd8f67f4cb981e (DEFAULT_TABLE table=default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.867820 1157 ts_tablet_manager.cc:1918] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.868472 1157 log.cc:1199] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/7ae96f2da36744198e3d38cb07bb47a0
W20250629 01:51:42.871094 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 829c64e2c71347f4af2c8870cdd8af31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829c64e2c71347f4af2c8870cdd8af31 already in progress: creating tablet
I20250629 01:51:42.871114 1159 tablet_bootstrap.cc:492] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.871294 1159 ts_tablet_manager.cc:1397] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.001s
I20250629 01:51:42.872457 1156 tablet_replica.cc:331] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.872830 1156 raft_consensus.cc:2241] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.873224 1156 raft_consensus.cc:2270] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.873314 1159 raft_consensus.cc:357] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.873586 1159 raft_consensus.cc:383] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.873731 1159 raft_consensus.cc:738] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.874049 1159 consensus_queue.cc:260] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.873360 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 (DEFAULT_TABLE table=default.Table_f128352ab4bd4f258d90d9a4022577a8 [id=b516da4148384818bcc69e4259f20ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.871060 1157 ts_tablet_manager.cc:1939] T 7ae96f2da36744198e3d38cb07bb47a0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:42.874662 1156 ts_tablet_manager.cc:1905] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.873447 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 541728ff8dc3471ba90784b4a769c8e1 (DEFAULT_TABLE table=default.Table_92068d6a585a434da503666ff8ac9536 [id=fd0ea1adec3e487ab647680c75b7fde8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.875574 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 34172d05ae40417494cd8f67f4cb981e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.875936 1157 tablet_replica.cc:331] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.876077 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 (DEFAULT_TABLE table=default.Table_9b3067ec3f994a44bf4ecacbc937f414 [id=5d8dd5f1acfd4c57b7080fc663a1d32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.876438 1157 raft_consensus.cc:2241] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:42.877540 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.878082 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c071543aa01840ecaaa8d2af64bf15b7" } new_table_name: "default.Table_f1a2e572d6ab48b8a6f6a2b818324c38" modify_external_catalogs: true
I20250629 01:51:42.880532 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_33e66ecca3054e9fa4f978c37661c5d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.880956 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33e66ecca3054e9fa4f978c37661c5d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.881968 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 432680cae61946af9859760fe23110ec (DEFAULT_TABLE table=default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 [id=2dfa777b5eab4acb9adfd35ab56c17d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.882375 1364 raft_consensus.cc:491] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:42.882483 1364 raft_consensus.cc:513] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.882856 1156 ts_tablet_manager.cc:1918] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.882946 1156 log.cc:1199] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/64477468c6da4b4ba282d8530eb5cbb5
I20250629 01:51:42.883080 1364 leader_election.cc:290] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:42.883210 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "64477468c6da4b4ba282d8530eb5cbb5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:42.883499 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3ba0e267f48a4bf1b58fb2339694719f
I20250629 01:51:42.883530 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 35990d206ba64159ab05ab7b7b19d064 (DEFAULT_TABLE table=default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 [id=8d49193b68504e77818e373ecad346b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.883962 448 catalog_manager.cc:6652] ReplaceTablet: tablet 3ba0e267f48a4bf1b58fb2339694719f deleted and replaced by tablet f898483ab9514132be8537ad03720097
I20250629 01:51:42.888859 1159 ts_tablet_manager.cc:1428] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.017s user 0.001s sys 0.004s
I20250629 01:51:42.889353 1159 tablet_bootstrap.cc:492] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:42.889472 710 leader_election.cc:343] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:42.889623 710 leader_election.cc:304] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:42.889813 710 raft_consensus.cc:2684] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:42.889631 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f with delete_type TABLET_DATA_DELETED (replaced by tablet f898483ab9514132be8537ad03720097) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.891924 448 master_service.cc:948] ReplaceTablet: received request to replace tablet dd6013bb40c248f29f40eab2c3008f6b from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:42.892517 986 leader_election.cc:343] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: d1956051272e450ab2ba3322ba23ab69
I20250629 01:51:42.892707 1159 tablet_bootstrap.cc:654] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:42.892939 1156 ts_tablet_manager.cc:1939] T 64477468c6da4b4ba282d8530eb5cbb5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.893006 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 683c87ea2ad44cecb2366d4a846d9195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.884364 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 81b32bac5a2144d1943285f5f446d53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.894522 1430 consensus_queue.cc:1035] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:42.894943 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 64477468c6da4b4ba282d8530eb5cbb5 (table default.Table_6f9bb7305f9d42fbb5b12690468aed5f [id=81e46758eee54c7caae081f9232e9071]) successfully deleted
I20250629 01:51:42.894969 1339 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:42.895303 1156 tablet_replica.cc:331] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.895538 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0c712d978a3848b9bb3abdbce8314dec
I20250629 01:51:42.895726 1156 raft_consensus.cc:2241] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.895869 1156 raft_consensus.cc:2270] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.895969 447 catalog_manager.cc:6652] ReplaceTablet: tablet 0c712d978a3848b9bb3abdbce8314dec deleted and replaced by tablet 0323947b135a4308903b1eed1ca4db71
I20250629 01:51:42.896734 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c712d978a3848b9bb3abdbce8314dec with delete_type TABLET_DATA_DELETED (replaced by tablet 0323947b135a4308903b1eed1ca4db71) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.900705 1156 ts_tablet_manager.cc:1905] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.903285 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432680cae61946af9859760fe23110ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.904799 1158 raft_consensus.cc:2241] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:42.911284 1156 ts_tablet_manager.cc:1918] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:42.914186 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 86832a3c26224c7ea0b9b4b6866d3880
I20250629 01:51:42.917176 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f with delete_type TABLET_DATA_DELETED (replaced by tablet f898483ab9514132be8537ad03720097) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.917764 1159 tablet_bootstrap.cc:492] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:42.917987 1159 ts_tablet_manager.cc:1397] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.014s
I20250629 01:51:42.919049 1159 raft_consensus.cc:357] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.919304 1159 raft_consensus.cc:383] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.919409 1159 raft_consensus.cc:738] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:42.919821 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 351f4a7ae7e244999dda81d282de505d
I20250629 01:51:42.919778 1159 consensus_queue.cc:260] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.920207 446 catalog_manager.cc:6652] ReplaceTablet: tablet 351f4a7ae7e244999dda81d282de505d deleted and replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b
I20250629 01:51:42.920509 1159 ts_tablet_manager.cc:1428] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:42.920961 1159 tablet_bootstrap.cc:492] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:42.922492 1156 log.cc:1199] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/2d44151acb0d4bf4a630c086b1ac8ff7
I20250629 01:51:42.915555 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.921058 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 351f4a7ae7e244999dda81d282de505d with delete_type TABLET_DATA_DELETED (replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:42.924664 1156 ts_tablet_manager.cc:1939] T 2d44151acb0d4bf4a630c086b1ac8ff7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:42.929064 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 (DEFAULT_TABLE table=default.Table_821e341e379740a59615270511038004 [id=910c490dcc294f58b93e5b6c4846d263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.930080 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]) successfully deleted
I20250629 01:51:42.930727 1159 tablet_bootstrap.cc:654] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:42.932332 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b7c7f2ac44734a39ab6434a74c91b9e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c7f2ac44734a39ab6434a74c91b9e4 already in progress: creating tablet
W20250629 01:51:42.932710 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 81b32bac5a2144d1943285f5f446d53d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81b32bac5a2144d1943285f5f446d53d already in progress: creating tablet
W20250629 01:51:42.932468 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 87849.1us
I20250629 01:51:42.932801 1157 pending_rounds.cc:70] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Trying to abort 2 pending ops.
I20250629 01:51:42.932880 1157 pending_rounds.cc:77] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7172759153947455488 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "2d44151acb0d4bf4a630c086b1ac8ff7" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_d219b9a9062745bca33e3a0fcf553be0" dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" new_extra_config { } }
W20250629 01:51:42.933174 1157 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:56376: Aborted: Op aborted
I20250629 01:51:42.933434 1157 pending_rounds.cc:77] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759150706311168 op_type: NO_OP noop_request { }
I20250629 01:51:42.933542 1157 raft_consensus.cc:2887] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:42.933638 1157 raft_consensus.cc:2270] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.934376 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1956051272e450ab2ba3322ba23ab69" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:42.934569 916 raft_consensus.cc:2466] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:42.934697 1157 ts_tablet_manager.cc:1905] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.935129 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 5e9f6de505a34afbae48a373b4b2a0cf from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:42.936749 891 tablet_service.cc:1468] Processing CreateTablet for tablet 013ab393a0f14bdc89389349bcc5d24b (DEFAULT_TABLE table=default.Table_f248b655abe641dbbf9eb656ff017e53 [id=55cb83afbc4f461f8946c50d83054bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.937116 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 013ab393a0f14bdc89389349bcc5d24b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:42.937306 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 34172d05ae40417494cd8f67f4cb981e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34172d05ae40417494cd8f67f4cb981e already in progress: creating tablet
I20250629 01:51:42.937876 1156 tablet_replica.cc:331] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:42.938278 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 7ae96f2da36744198e3d38cb07bb47a0 (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:42.938659 1156 raft_consensus.cc:2241] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.938977 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.939325 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 351f4a7ae7e244999dda81d282de505d with delete_type TABLET_DATA_DELETED (replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.939378 1156 raft_consensus.cc:2270] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.940251 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_96f437ef98bb46f69002f7508991afbe" } new_table_name: "default.Table_e69e519c24094e298cdcd58c4567f525" modify_external_catalogs: true
I20250629 01:51:42.940514 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f with delete_type TABLET_DATA_DELETED (replaced by tablet f898483ab9514132be8537ad03720097) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.941005 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c712d978a3848b9bb3abdbce8314dec with delete_type TABLET_DATA_DELETED (replaced by tablet 0323947b135a4308903b1eed1ca4db71) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:42.941134 894 tablet_service.cc:1468] Processing CreateTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 (DEFAULT_TABLE table=default.Table_9b3067ec3f994a44bf4ecacbc937f414 [id=5d8dd5f1acfd4c57b7080fc663a1d32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.941295 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f248b655abe641dbbf9eb656ff017e53" } new_table_name: "default.Table_a51f9e473f8f426db01a3dbfd4303993" modify_external_catalogs: true
I20250629 01:51:42.941447 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ce1bd4af434f7e99092aafc0ab0118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.942667 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_177db27d5e8b4755ad499157679a6f95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:42.943114 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_177db27d5e8b4755ad499157679a6f95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:42.944154 886 tablet_service.cc:1468] Processing CreateTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 (DEFAULT_TABLE table=default.Table_821e341e379740a59615270511038004 [id=910c490dcc294f58b93e5b6c4846d263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.944247 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14b2bea86460403f89586e3ebdbfd084" } modify_external_catalogs: true
I20250629 01:51:42.944396 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14b2bea86460403f89586e3ebdbfd084" } modify_external_catalogs: true
I20250629 01:51:42.944499 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196f1bd233ab4399a9d22a55b8bc2635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.949198 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 78a5f38231a34efaab177a23cf693d51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:42.957851 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 990d1fcdac1c406facb2da722421d7f4 consensus state has no leader
I20250629 01:51:42.963541 1157 ts_tablet_manager.cc:1918] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:42.963641 1157 log.cc:1199] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/2d44151acb0d4bf4a630c086b1ac8ff7
I20250629 01:51:42.964005 1157 ts_tablet_manager.cc:1939] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:42.965219 1157 tablet_replica.cc:331] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:42.965425 1157 raft_consensus.cc:2241] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:42.965561 1157 raft_consensus.cc:2270] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:42.966061 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.967082 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]) successfully deleted
I20250629 01:51:42.968766 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a23d7cbbf7804d2e95f76b62ec45c158" } modify_external_catalogs: true
I20250629 01:51:42.968940 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a23d7cbbf7804d2e95f76b62ec45c158" } modify_external_catalogs: true
I20250629 01:51:42.970388 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2bc100d78a348c88ed48ed66870200b" } modify_external_catalogs: true
I20250629 01:51:42.970501 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2bc100d78a348c88ed48ed66870200b" } modify_external_catalogs: true
I20250629 01:51:42.971235 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3abcbdb38a38484782a28269e7b69beb" } modify_external_catalogs: true
I20250629 01:51:42.971354 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3abcbdb38a38484782a28269e7b69beb" } modify_external_catalogs: true
I20250629 01:51:42.972222 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 351f4a7ae7e244999dda81d282de505d with delete_type TABLET_DATA_DELETED (replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.975441 757 tablet_service.cc:1468] Processing CreateTablet for tablet e83009c1599a43d18d1c73c0101ab0be (DEFAULT_TABLE table=default.Table_6b7f597bfe3947f884cf5e129049c99b [id=e4518081a6904de8a019b6d2d015d7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.975796 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83009c1599a43d18d1c73c0101ab0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.984946 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e83009c1599a43d18d1c73c0101ab0be (DEFAULT_TABLE table=default.Table_6b7f597bfe3947f884cf5e129049c99b [id=e4518081a6904de8a019b6d2d015d7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.986697 754 tablet_service.cc:1468] Processing CreateTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 (DEFAULT_TABLE table=default.Table_9b82f19de1c74fcd9973b4a4084712e1 [id=a9617884dddf40ee883c482c170d8a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.987036 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 241a5c9aee3f4297b67b18380ef5ef88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.992482 1154 tablet_bootstrap.cc:492] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:42.992550 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541728ff8dc3471ba90784b4a769c8e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.992641 1154 ts_tablet_manager.cc:1397] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.354s user 0.009s sys 0.006s
I20250629 01:51:42.993355 1154 raft_consensus.cc:357] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:42.993518 1154 raft_consensus.cc:383] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:42.993584 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83009c1599a43d18d1c73c0101ab0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:42.996155 1158 pending_rounds.cc:70] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:42.996222 1158 pending_rounds.cc:77] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759153273184256 op_type: NO_OP noop_request { }
I20250629 01:51:42.996335 1158 raft_consensus.cc:2887] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:42.996429 1158 raft_consensus.cc:2270] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:42.996637 1413 log.cc:927] Time spent T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20250629 01:51:42.996608 988 leader_election.cc:304] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:42.996852 1311 raft_consensus.cc:2802] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250629 01:51:42.996901 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 61159.3us
I20250629 01:51:42.996945 1311 raft_consensus.cc:491] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:42.992501 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c712d978a3848b9bb3abdbce8314dec with delete_type TABLET_DATA_DELETED (replaced by tablet 0323947b135a4308903b1eed1ca4db71) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:42.997519 1311 raft_consensus.cc:3058] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:42.997519 1413 consensus_queue.cc:1162] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4712643b602e4f898752fcab6a1e5913" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250629 01:51:42.998147 1158 ts_tablet_manager.cc:1905] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:42.998746 1387 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:43.000237 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ce1bd4af434f7e99092aafc0ab0118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.001431 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c61ad1c9857141faad549e9cc863671e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.001924 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c61ad1c9857141faad549e9cc863671e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.003425 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7768d2d7c9814ba48196e54d12a630a5" } new_table_name: "default.Table_4364b35675354c158cf8825e27a2ee9a" modify_external_catalogs: true
I20250629 01:51:42.993602 1154 raft_consensus.cc:738] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.004202 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35990d206ba64159ab05ab7b7b19d064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.004279 1154 consensus_queue.cc:260] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.005115 1311 raft_consensus.cc:513] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.005970 1311 leader_election.cc:290] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:43.006768 1154 ts_tablet_manager.cc:1428] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
W20250629 01:51:43.006986 986 leader_election.cc:343] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: d1956051272e450ab2ba3322ba23ab69
I20250629 01:51:43.007426 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 (DEFAULT_TABLE table=default.Table_9b82f19de1c74fcd9973b4a4084712e1 [id=a9617884dddf40ee883c482c170d8a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.008181 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 241a5c9aee3f4297b67b18380ef5ef88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.008544 757 tablet_service.cc:1468] Processing CreateTablet for tablet b6fa908ded8240bb962674e1efb29c74 (DEFAULT_TABLE table=default.Table_d9c78256954c4e8191c1373909f6b086 [id=9e20604c9f594334b255b4ca0a36012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.008846 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fa908ded8240bb962674e1efb29c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.014504 1013 tablet_service.cc:1468] Processing CreateTablet for tablet b6fa908ded8240bb962674e1efb29c74 (DEFAULT_TABLE table=default.Table_d9c78256954c4e8191c1373909f6b086 [id=9e20604c9f594334b255b4ca0a36012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:42.929889 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 013ab393a0f14bdc89389349bcc5d24b (DEFAULT_TABLE table=default.Table_f248b655abe641dbbf9eb656ff017e53 [id=55cb83afbc4f461f8946c50d83054bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.022172 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1956051272e450ab2ba3322ba23ab69" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:43.022352 916 raft_consensus.cc:3058] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.024273 1311 raft_consensus.cc:491] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.025249 916 raft_consensus.cc:2466] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:43.025938 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e962a020664b42ea918e5952dc43aade. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.026731 988 leader_election.cc:304] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:43.026960 1364 raft_consensus.cc:2802] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:43.027320 754 tablet_service.cc:1468] Processing CreateTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 (DEFAULT_TABLE table=default.Table_3d334b0f295e42378d74e5e746d911e2 [id=b890fb8e74dc464bbf195cdfa89814be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.007027 1154 tablet_bootstrap.cc:492] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:43.028105 1382 tablet.cc:1722] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:43.027662 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.030229 1156 ts_tablet_manager.cc:1905] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.031551 1155 tablet_bootstrap.cc:492] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.031693 1155 ts_tablet_manager.cc:1397] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.194s user 0.002s sys 0.003s
I20250629 01:51:43.033071 1364 raft_consensus.cc:695] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:43.033556 777 raft_consensus.cc:1273] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:43.035018 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d925ffce89b40258b29ec021bfa5f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.035435 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 943f584fb9554fa4b65e5302443cec92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.035239 1350 consensus_queue.cc:1035] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:43.035331 1364 consensus_queue.cc:237] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.036705 1049 raft_consensus.cc:1273] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:43.037225 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942f7bbe29f94083b2494aab8fbcc0c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.038105 1350 consensus_queue.cc:1035] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:43.042943 1158 ts_tablet_manager.cc:1918] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:43.041993 1311 raft_consensus.cc:513] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.044227 1023 tablet_service.cc:1468] Processing CreateTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 (DEFAULT_TABLE table=default.Table_3d334b0f295e42378d74e5e746d911e2 [id=b890fb8e74dc464bbf195cdfa89814be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.045256 891 tablet_service.cc:1468] Processing CreateTablet for tablet e83009c1599a43d18d1c73c0101ab0be (DEFAULT_TABLE table=default.Table_6b7f597bfe3947f884cf5e129049c99b [id=e4518081a6904de8a019b6d2d015d7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.045609 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83009c1599a43d18d1c73c0101ab0be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.046083 1158 log.cc:1199] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0c28ea445cec4b35be800dbc8c37af7c
I20250629 01:51:43.046207 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34172d05ae40417494cd8f67f4cb981e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.046777 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a485e0e6e2f04aa6bd67a62f6d30be26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.046993 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47d4287317ec4788bd96ad174c31748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.048223 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.050660 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fa908ded8240bb962674e1efb29c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.051308 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5a1e71ab9a84caea5c917d422336bc8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:43.051788 986 leader_election.cc:343] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:43.052330 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5a1e71ab9a84caea5c917d422336bc8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:43.052832 988 leader_election.cc:343] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:43.054610 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 34172d05ae40417494cd8f67f4cb981e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34172d05ae40417494cd8f67f4cb981e already in progress: creating tablet
I20250629 01:51:43.054754 886 tablet_service.cc:1468] Processing CreateTablet for tablet b6fa908ded8240bb962674e1efb29c74 (DEFAULT_TABLE table=default.Table_d9c78256954c4e8191c1373909f6b086 [id=9e20604c9f594334b255b4ca0a36012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.055116 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fa908ded8240bb962674e1efb29c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.055346 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 196f1bd233ab4399a9d22a55b8bc2635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.055593 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8799b1858bf2408a85f0956578639393 (DEFAULT_TABLE table=default.Table_1b80dba0f8d04043b95a7aaa176636b9 [id=511866fda7ae4bde9cea9777b36cd76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.055683 1157 ts_tablet_manager.cc:1905] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.055884 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8799b1858bf2408a85f0956578639393. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.058223 988 leader_election.cc:304] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:43.058503 1383 raft_consensus.cc:2747] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:43.059391 1311 leader_election.cc:290] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:43.059741 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 013ab393a0f14bdc89389349bcc5d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.061085 1158 ts_tablet_manager.cc:1939] T 0c28ea445cec4b35be800dbc8c37af7c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:43.061542 894 tablet_service.cc:1468] Processing CreateTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 (DEFAULT_TABLE table=default.Table_3d334b0f295e42378d74e5e746d911e2 [id=b890fb8e74dc464bbf195cdfa89814be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.061842 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.062841 891 tablet_service.cc:1468] Processing CreateTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 (DEFAULT_TABLE table=default.Table_9b82f19de1c74fcd9973b4a4084712e1 [id=a9617884dddf40ee883c482c170d8a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.063179 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 241a5c9aee3f4297b67b18380ef5ef88. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.066398 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b7c7f2ac44734a39ab6434a74c91b9e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c7f2ac44734a39ab6434a74c91b9e4 already in progress: creating tablet
W20250629 01:51:43.069778 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 81b32bac5a2144d1943285f5f446d53d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81b32bac5a2144d1943285f5f446d53d already in progress: creating tablet
I20250629 01:51:43.070058 1158 tablet_replica.cc:331] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:43.070248 1158 raft_consensus.cc:2241] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:43.070488 1158 pending_rounds.cc:70] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:43.070550 1158 pending_rounds.cc:77] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759154836283392 op_type: NO_OP noop_request { }
I20250629 01:51:43.070662 1158 raft_consensus.cc:2887] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:43.070768 1158 raft_consensus.cc:2270] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.039314 1155 raft_consensus.cc:357] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.071223 1155 raft_consensus.cc:383] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.071295 1155 raft_consensus.cc:738] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.071530 1155 consensus_queue.cc:260] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.071851 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 (DEFAULT_TABLE table=default.Table_a47465ab39a14717b174e77b563b4100 [id=af75db4dc824485997f324795f65607b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.071385 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0c28ea445cec4b35be800dbc8c37af7c (table default.Table_7a50e6b289ed401092480b4d315d2840 [id=c1e051380c514aa59fd3d4d84ee465d4]) successfully deleted
I20250629 01:51:43.074815 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 8799b1858bf2408a85f0956578639393 (DEFAULT_TABLE table=default.Table_1b80dba0f8d04043b95a7aaa176636b9 [id=511866fda7ae4bde9cea9777b36cd76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.075278 1155 ts_tablet_manager.cc:1428] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20250629 01:51:43.078101 1159 tablet_bootstrap.cc:492] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.078253 1159 ts_tablet_manager.cc:1397] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.157s user 0.036s sys 0.023s
I20250629 01:51:43.079090 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd084685cc0407da9e12b0a4df386e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.079103 1159 raft_consensus.cc:357] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.079295 1159 raft_consensus.cc:383] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.079401 1159 raft_consensus.cc:738] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.079671 1159 consensus_queue.cc:260] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.080248 1159 ts_tablet_manager.cc:1428] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:43.080758 1159 tablet_bootstrap.cc:492] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.082767 1159 tablet_bootstrap.cc:654] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.084263 1155 tablet_bootstrap.cc:492] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.092263 891 tablet_service.cc:1468] Processing CreateTablet for tablet 8799b1858bf2408a85f0956578639393 (DEFAULT_TABLE table=default.Table_1b80dba0f8d04043b95a7aaa176636b9 [id=511866fda7ae4bde9cea9777b36cd76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.092725 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8799b1858bf2408a85f0956578639393. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.093211 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 (DEFAULT_TABLE table=default.Table_a47465ab39a14717b174e77b563b4100 [id=af75db4dc824485997f324795f65607b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.093699 894 tablet_service.cc:1468] Processing CreateTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 (DEFAULT_TABLE table=default.Table_a47465ab39a14717b174e77b563b4100 [id=af75db4dc824485997f324795f65607b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.094066 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd084685cc0407da9e12b0a4df386e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.095574 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8799b1858bf2408a85f0956578639393. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.097036 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fca257d6cb614110948e1045dee44bdc consensus state has no leader
I20250629 01:51:43.100291 1339 tablet.cc:1722] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:43.100920 1382 tablet.cc:1722] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:51:43.112809 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 013ab393a0f14bdc89389349bcc5d24b consensus state has no leader
I20250629 01:51:43.113209 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_084d0d93b035470e8e5dbd670d05c9e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.114411 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_084d0d93b035470e8e5dbd670d05c9e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.121389 1155 tablet_bootstrap.cc:654] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.121724 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_328919206d1e44838bc19f8a2ff9d371 [id=ac80888442774af495a53bba44dfa763] alter complete (version 4)
I20250629 01:51:43.122400 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0072df8d087f4e8e9541d7f90a27bf4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.123003 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0072df8d087f4e8e9541d7f90a27bf4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.126060 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d1956051272e450ab2ba3322ba23ab69 with delete_type TABLET_DATA_DELETED (replaced by tablet 02d072bf72044dc48d8090660ecb7d47) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:43.134500 1435 consensus_queue.cc:1162] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4712643b602e4f898752fcab6a1e5913" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250629 01:51:43.141870 1158 ts_tablet_manager.cc:1905] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.142825 1263 consensus_queue.cc:1035] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:43.145282 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dd084685cc0407da9e12b0a4df386e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.125375 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_165130da0f884c2f99cc4de1d8689cd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.150441 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e82b1352bb4d49769808dc402c85890f
I20250629 01:51:43.162943 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet fb359994b20748518a676045ac6f0df8
I20250629 01:51:43.164516 1157 ts_tablet_manager.cc:1918] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.164640 1157 log.cc:1199] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/97e03c52aeb4432cab002abdae5109ba
I20250629 01:51:43.164821 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f898483ab9514132be8537ad03720097
I20250629 01:51:43.164937 1157 ts_tablet_manager.cc:1939] T 97e03c52aeb4432cab002abdae5109ba P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.165167 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f220872b43b3433694c2f522f6e54acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.165846 1157 tablet_replica.cc:331] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.165951 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet be8e0946a20340b38835c4d8daeaa7b8
I20250629 01:51:43.166019 1157 raft_consensus.cc:2241] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.167009 1156 ts_tablet_manager.cc:1918] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.167044 1159 tablet_bootstrap.cc:492] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.167120 1156 log.cc:1199] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/c5a1e71ab9a84caea5c917d422336bc8
I20250629 01:51:43.167187 1159 ts_tablet_manager.cc:1397] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.000s
I20250629 01:51:43.167436 1156 ts_tablet_manager.cc:1939] T c5a1e71ab9a84caea5c917d422336bc8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:43.167902 757 tablet_service.cc:1515] Processing DeleteTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.167974 891 tablet_service.cc:1515] Processing DeleteTablet for tablet fb359994b20748518a676045ac6f0df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.168165 886 tablet_service.cc:1515] Processing DeleteTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.168319 1159 raft_consensus.cc:357] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.168625 1159 raft_consensus.cc:383] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.168747 1159 raft_consensus.cc:738] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.169024 1159 consensus_queue.cc:260] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.170122 1159 ts_tablet_manager.cc:1428] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:43.170802 1159 tablet_bootstrap.cc:492] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.170872 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bba7ed21c41c46ca8f68ce01c5225b1e" } new_table_name: "default.Table_05483b691b564e7482b9d04d0afb1b29" modify_external_catalogs: true
I20250629 01:51:43.172816 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7253ebd64bc14d1c89a2602b5f64e13a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.173398 755 tablet_service.cc:1515] Processing DeleteTablet for tablet fb359994b20748518a676045ac6f0df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.174059 1159 tablet_bootstrap.cc:654] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.174504 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet c5a1e71ab9a84caea5c917d422336bc8 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
W20250629 01:51:43.175122 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f220872b43b3433694c2f522f6e54acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.179572 1387 tablet.cc:1722] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:51:43.182615 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_165130da0f884c2f99cc4de1d8689cd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.183334 1157 raft_consensus.cc:2270] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.184042 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 97e03c52aeb4432cab002abdae5109ba (table default.Table_4b11b5c3ee064217b3fcbeb9ad01293c [id=7520c37ee81241f7a7bcd8d17ddab3a0]) successfully deleted
I20250629 01:51:43.184445 1157 ts_tablet_manager.cc:1905] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:43.184960 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dfe921a84694903bbdb3fed3d6f2761 consensus state has no leader
I20250629 01:51:43.186163 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_bba7ed21c41c46ca8f68ce01c5225b1e [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 2 got 0
I20250629 01:51:43.187554 1155 tablet_bootstrap.cc:492] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.187714 1155 ts_tablet_manager.cc:1397] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.104s user 0.004s sys 0.004s
I20250629 01:51:43.188326 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 05905638c6e241a095438f277f53adab from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.188591 1155 raft_consensus.cc:357] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.188738 1155 raft_consensus.cc:383] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.188819 1155 raft_consensus.cc:738] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.188803 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fed4bed522aa4c8192e79303877c5472"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.189023 1155 consensus_queue.cc:260] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.189462 1155 ts_tablet_manager.cc:1428] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:43.189791 1155 tablet_bootstrap.cc:492] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.191318 1155 tablet_bootstrap.cc:654] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:43.192670 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fed4bed522aa4c8192e79303877c5472 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.195588 1157 ts_tablet_manager.cc:1918] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.195706 1157 log.cc:1199] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/78a5f38231a34efaab177a23cf693d51
I20250629 01:51:43.195991 1159 tablet_bootstrap.cc:492] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.196056 1157 ts_tablet_manager.cc:1939] T 78a5f38231a34efaab177a23cf693d51 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.196589 1159 ts_tablet_manager.cc:1397] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
W20250629 01:51:43.196674 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 86832a3c26224c7ea0b9b4b6866d3880 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86832a3c26224c7ea0b9b4b6866d3880 already in progress: creating tablet
W20250629 01:51:43.196905 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 351f4a7ae7e244999dda81d282de505d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351f4a7ae7e244999dda81d282de505d already in progress: creating tablet
I20250629 01:51:43.196964 1157 tablet_replica.cc:331] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.197150 1157 raft_consensus.cc:2241] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.197304 1157 raft_consensus.cc:2270] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:43.197505 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0c712d978a3848b9bb3abdbce8314dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c712d978a3848b9bb3abdbce8314dec already in progress: creating tablet
I20250629 01:51:43.197716 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 78a5f38231a34efaab177a23cf693d51 (table default.Table_601ff9b6b68b4acb9206a64afa4339a8 [id=486534f71b544cee8fd56472d6806f80]) successfully deleted
I20250629 01:51:43.198093 1159 raft_consensus.cc:357] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.198211 1155 tablet_bootstrap.cc:492] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.198240 1159 raft_consensus.cc:383] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.198309 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e82b1352bb4d49769808dc402c85890f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.198346 1155 ts_tablet_manager.cc:1397] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20250629 01:51:43.198331 1159 raft_consensus.cc:738] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.199098 1263 raft_consensus.cc:491] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.199226 1263 raft_consensus.cc:513] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.199182 1159 consensus_queue.cc:260] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.199831 1263 leader_election.cc:290] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:43.200132 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0c712d978a3848b9bb3abdbce8314dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c712d978a3848b9bb3abdbce8314dec already in progress: creating tablet
W20250629 01:51:43.200328 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 86832a3c26224c7ea0b9b4b6866d3880 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86832a3c26224c7ea0b9b4b6866d3880 already in progress: creating tablet
I20250629 01:51:43.200407 1159 ts_tablet_manager.cc:1428] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20250629 01:51:43.200505 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 351f4a7ae7e244999dda81d282de505d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351f4a7ae7e244999dda81d282de505d already in progress: creating tablet
I20250629 01:51:43.200635 1159 tablet_bootstrap.cc:492] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:43.200724 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet be8e0946a20340b38835c4d8daeaa7b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8e0946a20340b38835c4d8daeaa7b8 already in progress: creating tablet
W20250629 01:51:43.200934 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet fb359994b20748518a676045ac6f0df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb359994b20748518a676045ac6f0df8 already in progress: creating tablet
I20250629 01:51:43.201284 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a229b4f9d19147a5990f799d1dc0e5d7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:43.201921 712 leader_election.cc:343] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.202898 1155 raft_consensus.cc:357] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.203101 1155 raft_consensus.cc:383] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:51:43.203315 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7253ebd64bc14d1c89a2602b5f64e13a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.203177 1155 raft_consensus.cc:738] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.204116 1155 consensus_queue.cc:260] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.204279 1159 tablet_bootstrap.cc:654] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.204952 1155 ts_tablet_manager.cc:1428] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250629 01:51:43.205799 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_acc52ee4aee240dc82efdc80f09faa91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.206192 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc52ee4aee240dc82efdc80f09faa91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.206426 1155 tablet_bootstrap.cc:492] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.209239 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e82b1352bb4d49769808dc402c85890f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.209412 1157 ts_tablet_manager.cc:1905] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.209941 1159 tablet_bootstrap.cc:492] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
W20250629 01:51:43.209975 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e82b1352bb4d49769808dc402c85890f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e82b1352bb4d49769808dc402c85890f already in progress: creating tablet
I20250629 01:51:43.210081 1159 ts_tablet_manager.cc:1397] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20250629 01:51:43.210889 1159 raft_consensus.cc:357] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.211102 1159 raft_consensus.cc:383] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.211161 1159 raft_consensus.cc:738] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.211205 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet e82b1352bb4d49769808dc402c85890f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.211400 1159 consensus_queue.cc:260] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.211660 1155 tablet_bootstrap.cc:654] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.212409 1159 ts_tablet_manager.cc:1428] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:43.212731 1159 tablet_bootstrap.cc:492] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.214905 1154 tablet_bootstrap.cc:654] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.215232 1159 tablet_bootstrap.cc:654] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.215399 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.215529 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet fb359994b20748518a676045ac6f0df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.216039 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a229b4f9d19147a5990f799d1dc0e5d7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:43.216234 1050 raft_consensus.cc:2466] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:43.216650 710 leader_election.cc:304] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:43.217023 1263 raft_consensus.cc:2802] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.217099 1263 raft_consensus.cc:491] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.217178 1263 raft_consensus.cc:3058] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.219489 1263 raft_consensus.cc:513] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.220235 1155 tablet_bootstrap.cc:492] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.220345 1155 ts_tablet_manager.cc:1397] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20250629 01:51:43.220973 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a229b4f9d19147a5990f799d1dc0e5d7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:43.220954 1155 raft_consensus.cc:357] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.221091 1155 raft_consensus.cc:383] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.221114 916 raft_consensus.cc:3058] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.221164 1155 raft_consensus.cc:738] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.221323 1155 consensus_queue.cc:260] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.222258 1159 tablet_bootstrap.cc:492] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.222381 1159 ts_tablet_manager.cc:1397] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250629 01:51:43.223035 1159 raft_consensus.cc:357] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.223202 1159 raft_consensus.cc:383] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.223276 1159 raft_consensus.cc:738] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.223990 1159 consensus_queue.cc:260] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.224607 1159 ts_tablet_manager.cc:1428] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:43.224895 1159 tablet_bootstrap.cc:492] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.226438 1159 tablet_bootstrap.cc:654] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.229897 1155 ts_tablet_manager.cc:1428] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20250629 01:51:43.230191 1155 tablet_bootstrap.cc:492] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.231689 1155 tablet_bootstrap.cc:654] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.232625 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_399e960d59fe4143843457bd372dbf63" } modify_external_catalogs: true
I20250629 01:51:43.232743 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_399e960d59fe4143843457bd372dbf63" } modify_external_catalogs: true
I20250629 01:51:43.233533 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 9cf553b291854a6f9e7ed78b5db5f18c from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.233870 1159 tablet_bootstrap.cc:492] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.233899 450 master_service.cc:948] ReplaceTablet: received request to replace tablet a2a9664bb7a946c48cf79a09955ecad5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.234025 1159 ts_tablet_manager.cc:1397] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20250629 01:51:43.234333 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6991a3825f4547818c6b688dfc8ff59c" } new_table_name: "default.Table_c320a3e0f5fb4ee782878b813157b1cb" modify_external_catalogs: true
I20250629 01:51:43.234802 1159 raft_consensus.cc:357] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.234997 1159 raft_consensus.cc:383] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.235069 1159 raft_consensus.cc:738] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.235296 1159 consensus_queue.cc:260] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.235383 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 0d925ffce89b40258b29ec021bfa5f92 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.235833 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7f1d914328fb40dc9df62f6057abd8e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.237082 1159 ts_tablet_manager.cc:1428] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:43.237404 1159 tablet_bootstrap.cc:492] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.238483 1157 ts_tablet_manager.cc:1918] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.238619 1157 log.cc:1199] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d1956051272e450ab2ba3322ba23ab69
I20250629 01:51:43.238926 1157 ts_tablet_manager.cc:1939] T d1956051272e450ab2ba3322ba23ab69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.239218 1159 tablet_bootstrap.cc:654] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.240382 452 master_service.cc:948] ReplaceTablet: received request to replace tablet a485e0e6e2f04aa6bd67a62f6d30be26 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.242771 1159 tablet_bootstrap.cc:492] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.242908 1159 ts_tablet_manager.cc:1397] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20250629 01:51:43.243129 916 raft_consensus.cc:2466] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:43.243530 712 leader_election.cc:304] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.243737 1159 raft_consensus.cc:357] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.243902 1159 raft_consensus.cc:383] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.243992 1159 raft_consensus.cc:738] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.245316 1155 tablet_bootstrap.cc:492] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.244213 1159 consensus_queue.cc:260] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.245450 1155 ts_tablet_manager.cc:1397] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20250629 01:51:43.245808 1159 ts_tablet_manager.cc:1428] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:43.246023 1159 tablet_bootstrap.cc:492] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.246059 1155 raft_consensus.cc:357] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.246619 1155 raft_consensus.cc:383] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.246685 1155 raft_consensus.cc:738] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.246869 1155 consensus_queue.cc:260] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.247296 1155 ts_tablet_manager.cc:1428] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:43.247543 1155 tablet_bootstrap.cc:492] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.247754 1159 tablet_bootstrap.cc:654] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.248428 757 tablet_service.cc:1468] Processing CreateTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f (DEFAULT_TABLE table=default.Table_7b31a72aec3c4710971cd48931f10395 [id=2fbd2c29b63c488ba11a1ed546f38da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.248795 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea681bd91e54e4898bcfb72bed6198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.249186 1155 tablet_bootstrap.cc:654] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.250221 1457 raft_consensus.cc:2802] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:43.251457 1457 raft_consensus.cc:695] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:43.252748 754 tablet_service.cc:1468] Processing CreateTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b (DEFAULT_TABLE table=default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.256419 1159 tablet_bootstrap.cc:492] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.257715 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d1956051272e450ab2ba3322ba23ab69 (table default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]) successfully deleted
W20250629 01:51:43.257930 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet fb359994b20748518a676045ac6f0df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb359994b20748518a676045ac6f0df8 already in progress: creating tablet
W20250629 01:51:43.258100 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet be8e0946a20340b38835c4d8daeaa7b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8e0946a20340b38835c4d8daeaa7b8 already in progress: creating tablet
W20250629 01:51:43.258257 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e82b1352bb4d49769808dc402c85890f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e82b1352bb4d49769808dc402c85890f already in progress: creating tablet
I20250629 01:51:43.258260 886 tablet_service.cc:1468] Processing CreateTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 (DEFAULT_TABLE table=default.Table_33e66ecca3054e9fa4f978c37661c5d2 [id=7edb7d438b5b4ab3b529cb31fc761610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.258553 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8b38c20eb1849b1a9f8acd40315eef5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.260345 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f1d914328fb40dc9df62f6057abd8e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.263870 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f (DEFAULT_TABLE table=default.Table_7b31a72aec3c4710971cd48931f10395 [id=2fbd2c29b63c488ba11a1ed546f38da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.264163 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea681bd91e54e4898bcfb72bed6198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.264561 888 tablet_service.cc:1468] Processing CreateTablet for tablet 0323947b135a4308903b1eed1ca4db71 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.264811 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0323947b135a4308903b1eed1ca4db71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.266700 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]). Expected version 1 got 0
I20250629 01:51:43.266908 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_05483b691b564e7482b9d04d0afb1b29 [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 3 got 2
I20250629 01:51:43.268323 887 tablet_service.cc:1468] Processing CreateTablet for tablet add482b533d045e7a3010064fae730cf (DEFAULT_TABLE table=default.Table_6991a3825f4547818c6b688dfc8ff59c [id=1f28cdb4b4d34d6cb5880828f17011ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.268589 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add482b533d045e7a3010064fae730cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.272188 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.272403 1157 tablet_replica.cc:331] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.272569 1157 raft_consensus.cc:2241] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.272712 1157 raft_consensus.cc:2270] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.272936 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.273322 1157 ts_tablet_manager.cc:1905] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.274094 1427 raft_consensus.cc:491] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.274212 1427 raft_consensus.cc:513] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.274693 1427 leader_election.cc:290] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.275278 885 tablet_service.cc:1468] Processing CreateTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b (DEFAULT_TABLE table=default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.275561 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897a4cefefe04f06b47070ec7fa1a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.279824 1157 ts_tablet_manager.cc:1918] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.279929 1157 log.cc:1199] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/3a45e95a4a6a45f9b7c7b09bd69112b4
I20250629 01:51:43.280253 1157 ts_tablet_manager.cc:1939] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.281906 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:43.282230 452 catalog_manager.cc:6652] ReplaceTablet: tablet a485e0e6e2f04aa6bd67a62f6d30be26 deleted and replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49
I20250629 01:51:43.282811 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0d925ffce89b40258b29ec021bfa5f92
I20250629 01:51:43.283144 448 catalog_manager.cc:6652] ReplaceTablet: tablet 0d925ffce89b40258b29ec021bfa5f92 deleted and replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b
I20250629 01:51:43.283829 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:43.284188 449 catalog_manager.cc:6652] ReplaceTablet: tablet 9cf553b291854a6f9e7ed78b5db5f18c deleted and replaced by tablet ddc24d0e668c49ac825b2578849ab2a3
I20250629 01:51:43.285032 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0a574f95ac2a4193a563e63cbe120768
I20250629 01:51:43.286965 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 0323947b135a4308903b1eed1ca4db71 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.287341 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0323947b135a4308903b1eed1ca4db71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.287442 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.287695 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a229b4f9d19147a5990f799d1dc0e5d7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:43.287837 1050 raft_consensus.cc:3058] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.287338 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b (DEFAULT_TABLE table=default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.289373 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897a4cefefe04f06b47070ec7fa1a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.256542 1159 ts_tablet_manager.cc:1397] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20250629 01:51:43.290958 1050 raft_consensus.cc:2466] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:43.291898 1159 raft_consensus.cc:357] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.292068 1159 raft_consensus.cc:383] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.292160 1159 raft_consensus.cc:738] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.292364 1159 consensus_queue.cc:260] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.292905 1159 ts_tablet_manager.cc:1428] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:43.293363 1159 tablet_bootstrap.cc:492] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.294888 1159 tablet_bootstrap.cc:654] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:43.295142 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add482b533d045e7a3010064fae730cf consensus state has no leader
I20250629 01:51:43.298857 1028 tablet_service.cc:1468] Processing CreateTablet for tablet add482b533d045e7a3010064fae730cf (DEFAULT_TABLE table=default.Table_6991a3825f4547818c6b688dfc8ff59c [id=1f28cdb4b4d34d6cb5880828f17011ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.299125 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add482b533d045e7a3010064fae730cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.301224 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f (DEFAULT_TABLE table=default.Table_7b31a72aec3c4710971cd48931f10395 [id=2fbd2c29b63c488ba11a1ed546f38da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.304466 1159 tablet_bootstrap.cc:492] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.304668 1159 ts_tablet_manager.cc:1397] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20250629 01:51:43.304816 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 (DEFAULT_TABLE table=default.Table_33e66ecca3054e9fa4f978c37661c5d2 [id=7edb7d438b5b4ab3b529cb31fc761610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.305668 1159 raft_consensus.cc:357] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.305874 1159 raft_consensus.cc:383] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.305965 1159 raft_consensus.cc:738] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.306203 1159 consensus_queue.cc:260] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.306805 1159 ts_tablet_manager.cc:1428] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:43.307050 1159 tablet_bootstrap.cc:492] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.308730 1159 tablet_bootstrap.cc:654] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.309532 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_33e66ecca3054e9fa4f978c37661c5d2" } new_table_name: "default.Table_d6eb1cb53be64d2d81c86f31b21bbecc" modify_external_catalogs: true
I20250629 01:51:43.312073 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 (table default.Table_0b441ce248f2448eabf2d7e3cddff976 [id=ba643eccd0e54ce083459bf535db1fe1]) successfully deleted
I20250629 01:51:43.313498 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 with delete_type TABLET_DATA_DELETED (replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.313773 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.314002 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8b38c20eb1849b1a9f8acd40315eef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.313990 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24d0e668c49ac825b2578849ab2a3) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.314215 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a574f95ac2a4193a563e63cbe120768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.315516 1339 tablet.cc:1722] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:43.315871 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:43.316002 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_05483b691b564e7482b9d04d0afb1b29 [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 3 got 2
I20250629 01:51:43.323536 1364 raft_consensus.cc:491] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.323697 1364 raft_consensus.cc:513] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.324872 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_175e2bb2b6c5433da19613f5f2e15823" } new_table_name: "default.Table_49846b0d606c4bf28681117f0a8340bd" modify_external_catalogs: true
I20250629 01:51:43.326728 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 with delete_type TABLET_DATA_DELETED (replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.331151 1311 raft_consensus.cc:491] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.331215 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5a1e71ab9a84caea5c917d422336bc8" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:43.331971 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6f6d8cf186fc4934ac5690bac0754373"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.332432 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f6d8cf186fc4934ac5690bac0754373 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.333192 1457 consensus_queue.cc:237] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.334739 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea681bd91e54e4898bcfb72bed6198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.335204 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0323947b135a4308903b1eed1ca4db71 (DEFAULT_TABLE table=default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.336380 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897a4cefefe04f06b47070ec7fa1a33b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.336908 1155 tablet_bootstrap.cc:492] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.337042 1155 ts_tablet_manager.cc:1397] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20250629 01:51:43.337896 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24d0e668c49ac825b2578849ab2a3) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:43.338016 850 leader_election.cc:343] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: c5a1e71ab9a84caea5c917d422336bc8
I20250629 01:51:43.338265 750 tablet_service.cc:1515] Processing DeleteTablet for tablet dd6013bb40c248f29f40eab2c3008f6b with delete_type TABLET_DATA_DELETED (replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.338358 1364 leader_election.cc:290] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:43.338407 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:43.338415 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a485e0e6e2f04aa6bd67a62f6d30be26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a485e0e6e2f04aa6bd67a62f6d30be26 already in progress: creating tablet
I20250629 01:51:43.338486 1156 tablet_replica.cc:331] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:43.338549 1263 leader_election.cc:290] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:43.338616 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0d925ffce89b40258b29ec021bfa5f92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d925ffce89b40258b29ec021bfa5f92 already in progress: creating tablet
W20250629 01:51:43.338779 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0a574f95ac2a4193a563e63cbe120768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a574f95ac2a4193a563e63cbe120768 already in progress: creating tablet
W20250629 01:51:43.338940 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9cf553b291854a6f9e7ed78b5db5f18c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf553b291854a6f9e7ed78b5db5f18c already in progress: creating tablet
I20250629 01:51:43.339648 755 tablet_service.cc:1468] Processing CreateTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 (DEFAULT_TABLE table=default.Table_33e66ecca3054e9fa4f978c37661c5d2 [id=7edb7d438b5b4ab3b529cb31fc761610]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:43.339691 988 leader_election.cc:343] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: 7ae96f2da36744198e3d38cb07bb47a0
I20250629 01:51:43.340257 747 tablet_service.cc:1468] Processing CreateTablet for tablet add482b533d045e7a3010064fae730cf (DEFAULT_TABLE table=default.Table_6991a3825f4547818c6b688dfc8ff59c [id=1f28cdb4b4d34d6cb5880828f17011ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.340797 1425 raft_consensus.cc:491] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.340924 1425 raft_consensus.cc:513] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.341051 1159 tablet_bootstrap.cc:492] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.341233 1159 ts_tablet_manager.cc:1397] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.007s
I20250629 01:51:43.341547 1425 leader_election.cc:290] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.342064 1159 raft_consensus.cc:357] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.342271 1159 raft_consensus.cc:383] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.342294 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 with delete_type TABLET_DATA_DELETED (replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.342339 1159 raft_consensus.cc:738] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.342563 1159 consensus_queue.cc:260] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.342796 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.343093 1159 ts_tablet_manager.cc:1428] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:43.343405 1159 tablet_bootstrap.cc:492] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.344702 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a574f95ac2a4193a563e63cbe120768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.344863 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0323947b135a4308903b1eed1ca4db71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.345041 1159 tablet_bootstrap.cc:654] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.345304 1048 raft_consensus.cc:2466] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:43.347229 1155 raft_consensus.cc:357] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.347376 1155 raft_consensus.cc:383] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.347738 1155 raft_consensus.cc:738] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.347934 1155 consensus_queue.cc:260] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.348608 1155 ts_tablet_manager.cc:1428] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250629 01:51:43.348865 1155 tablet_bootstrap.cc:492] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.349018 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24d0e668c49ac825b2578849ab2a3) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.349116 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet add482b533d045e7a3010064fae730cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.349335 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a574f95ac2a4193a563e63cbe120768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.347079 1382 tablet.cc:1722] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:43.344369 1465 consensus_queue.cc:1035] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20250629 01:51:43.351291 1159 tablet_bootstrap.cc:492] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.351336 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8b38c20eb1849b1a9f8acd40315eef5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.351401 986 leader_election.cc:343] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 7ae96f2da36744198e3d38cb07bb47a0
I20250629 01:51:43.351428 1159 ts_tablet_manager.cc:1397] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20250629 01:51:43.351490 986 leader_election.cc:304] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:43.352173 1159 raft_consensus.cc:357] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.352255 1364 raft_consensus.cc:2747] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:43.352335 1159 raft_consensus.cc:383] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.352423 1159 raft_consensus.cc:738] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.352627 1159 consensus_queue.cc:260] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.352721 850 leader_election.cc:304] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:43.353415 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a72c6040421642578fac307ea9914ef6 (table default.Table_05483b691b564e7482b9d04d0afb1b29 [id=6a41ce2e3dcf43ce9edb6392a244d9a2]). Expected version 3 got 2
I20250629 01:51:43.354607 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd6013bb40c248f29f40eab2c3008f6b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:43.354764 777 raft_consensus.cc:2466] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:43.355305 1350 raft_consensus.cc:2802] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.355402 1350 raft_consensus.cc:491] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.355486 1350 raft_consensus.cc:3058] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.356686 850 leader_election.cc:304] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.356902 1156 raft_consensus.cc:2241] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.357036 1156 raft_consensus.cc:2270] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.357280 1159 ts_tablet_manager.cc:1428] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:43.357506 1159 tablet_bootstrap.cc:492] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.358829 1159 tablet_bootstrap.cc:654] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.358860 1350 raft_consensus.cc:513] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.359431 1350 leader_election.cc:290] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.359611 1350 raft_consensus.cc:2802] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.359690 1350 raft_consensus.cc:491] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.359779 1350 raft_consensus.cc:3058] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:43.360780 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8b38c20eb1849b1a9f8acd40315eef5 consensus state has no leader
I20250629 01:51:43.361356 1156 ts_tablet_manager.cc:1905] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.355109 1311 raft_consensus.cc:513] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:43.362630 850 leader_election.cc:343] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: c5a1e71ab9a84caea5c917d422336bc8
I20250629 01:51:43.365939 1350 raft_consensus.cc:513] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.370555 1350 leader_election.cc:290] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.371807 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cf2dc25aa17a42fb91e6903713ca0d6c" } modify_external_catalogs: true
I20250629 01:51:43.371956 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cf2dc25aa17a42fb91e6903713ca0d6c" } modify_external_catalogs: true
I20250629 01:51:43.373119 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd6013bb40c248f29f40eab2c3008f6b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:43.373190 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7f1d914328fb40dc9df62f6057abd8e7" } modify_external_catalogs: true
I20250629 01:51:43.373320 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7f1d914328fb40dc9df62f6057abd8e7" } modify_external_catalogs: true
I20250629 01:51:43.374356 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c6e85ae2e824442fa3e307eda6dc1aa2" } modify_external_catalogs: true
I20250629 01:51:43.374490 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c6e85ae2e824442fa3e307eda6dc1aa2" } modify_external_catalogs: true
W20250629 01:51:43.375525 850 leader_election.cc:343] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:43.375862 1155 tablet_bootstrap.cc:654] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.380388 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "990d1fcdac1c406facb2da722421d7f4" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:43.380539 914 raft_consensus.cc:2466] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
W20250629 01:51:43.384434 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8976cca816b442cc9a7cb4f23cc13016 consensus state has no leader
I20250629 01:51:43.385735 888 tablet_service.cc:1515] Processing DeleteTablet for tablet e6b1ecf098ab4e498391e79cfb1c60b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.385951 1157 tablet_replica.cc:331] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.386046 887 tablet_service.cc:1515] Processing DeleteTablet for tablet c5a1e71ab9a84caea5c917d422336bc8 with delete_type TABLET_DATA_DELETED (replaced by tablet 97439c6a6e844610b55b899966ae5b67) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.386152 1157 raft_consensus.cc:2241] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.386256 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.386309 1157 raft_consensus.cc:2270] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.386437 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.385948 891 tablet_service.cc:1515] Processing DeleteTablet for tablet dd6013bb40c248f29f40eab2c3008f6b with delete_type TABLET_DATA_DELETED (replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.387019 1157 ts_tablet_manager.cc:1905] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.391278 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e1fe3880e51345269e852014d0d60bb6
I20250629 01:51:43.395987 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eb9a6975124a4d049fa2c7043760c952"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.396334 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1b80dba0f8d04043b95a7aaa176636b9" } modify_external_catalogs: true
W20250629 01:51:43.396492 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb9a6975124a4d049fa2c7043760c952 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.396467 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1b80dba0f8d04043b95a7aaa176636b9" } modify_external_catalogs: true
I20250629 01:51:43.399603 1157 ts_tablet_manager.cc:1918] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.399730 1157 log.cc:1199] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e6b1ecf098ab4e498391e79cfb1c60b0
I20250629 01:51:43.399765 891 tablet_service.cc:1515] Processing DeleteTablet for tablet e1fe3880e51345269e852014d0d60bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.400080 1157 ts_tablet_manager.cc:1939] T e6b1ecf098ab4e498391e79cfb1c60b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.401108 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e6b1ecf098ab4e498391e79cfb1c60b0 (table default.Table_7b25659fe37948109f17e9e6b15518cc [id=9a2d24af85fb40b3a4c49f37cca3ab62]) successfully deleted
I20250629 01:51:43.401603 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4364b35675354c158cf8825e27a2ee9a" } new_table_name: "default.Table_6ada0762e0224924acf20a4507ad2fa6" modify_external_catalogs: true
I20250629 01:51:43.401684 1157 tablet_replica.cc:331] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.403892 1157 raft_consensus.cc:2241] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.404402 1157 raft_consensus.cc:2270] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.405145 454 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 3131e12612ff488289e9fb62d16ed28e
I20250629 01:51:43.409431 1156 ts_tablet_manager.cc:1918] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.409904 1157 ts_tablet_manager.cc:1905] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.410892 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd6013bb40c248f29f40eab2c3008f6b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:43.411134 1387 tablet.cc:1722] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:43.411484 1156 log.cc:1199] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/dd6013bb40c248f29f40eab2c3008f6b
I20250629 01:51:43.411515 988 leader_election.cc:304] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:43.411754 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd6013bb40c248f29f40eab2c3008f6b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:43.412518 1364 raft_consensus.cc:2802] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.412629 1364 raft_consensus.cc:491] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.412734 1364 raft_consensus.cc:3058] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.413810 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet e1fe3880e51345269e852014d0d60bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.414144 1156 ts_tablet_manager.cc:1939] T dd6013bb40c248f29f40eab2c3008f6b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:43.414187 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5a1e71ab9a84caea5c917d422336bc8" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:43.415050 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet dd6013bb40c248f29f40eab2c3008f6b (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:51:43.415252 1050 raft_consensus.cc:3058] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:43.416873 850 leader_election.cc:343] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.417001 850 leader_election.cc:304] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:43.417433 850 leader_election.cc:343] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.417935 1430 raft_consensus.cc:2747] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250629 01:51:43.418071 1157 ts_tablet_manager.cc:1918] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.418218 1157 log.cc:1199] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/c5a1e71ab9a84caea5c917d422336bc8
I20250629 01:51:43.418941 1156 tablet_replica.cc:331] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:43.419209 1157 ts_tablet_manager.cc:1939] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.419219 745 tablet_service.cc:1515] Processing DeleteTablet for tablet e1fe3880e51345269e852014d0d60bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.419750 1157 tablet_replica.cc:331] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.419963 1157 raft_consensus.cc:2241] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.420122 1157 raft_consensus.cc:2270] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.421180 1159 tablet_bootstrap.cc:492] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.421350 1159 ts_tablet_manager.cc:1397] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.021s
I20250629 01:51:43.422175 1159 raft_consensus.cc:357] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.422416 1159 raft_consensus.cc:383] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.422422 454 master_service.cc:948] ReplaceTablet: received request to replace tablet f8b38c20eb1849b1a9f8acd40315eef5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.422493 1159 raft_consensus.cc:738] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.422736 1159 consensus_queue.cc:260] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.423966 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet c5a1e71ab9a84caea5c917d422336bc8 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
W20250629 01:51:43.424337 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5dfe921a84694903bbdb3fed3d6f2761 consensus state has no leader
I20250629 01:51:43.424376 1159 ts_tablet_manager.cc:1428] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:43.424616 1159 tablet_bootstrap.cc:492] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.426126 1159 tablet_bootstrap.cc:654] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.427474 1157 ts_tablet_manager.cc:1905] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.429656 1156 raft_consensus.cc:2241] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:43.430027 1156 pending_rounds.cc:70] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:43.430122 1156 pending_rounds.cc:77] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759156055511040 op_type: NO_OP noop_request { }
I20250629 01:51:43.430272 1156 raft_consensus.cc:2887] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:43.430397 1156 raft_consensus.cc:2270] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.432189 1311 leader_election.cc:290] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:43.432628 1472 consensus_queue.cc:1162] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250629 01:51:43.433290 1156 ts_tablet_manager.cc:1905] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.444957 1364 raft_consensus.cc:513] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.445461 1364 leader_election.cc:290] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:43.447932 1155 tablet_bootstrap.cc:492] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.448052 1155 ts_tablet_manager.cc:1397] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.099s user 0.000s sys 0.008s
I20250629 01:51:43.448670 1155 raft_consensus.cc:357] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.448827 1155 raft_consensus.cc:383] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.448894 1155 raft_consensus.cc:738] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.449108 1155 consensus_queue.cc:260] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.449508 1155 ts_tablet_manager.cc:1428] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:43.450515 1155 tablet_bootstrap.cc:492] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.452901 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5bb5e8cd427a4fdf9a1ccbc0fb4caaf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.453285 1155 tablet_bootstrap.cc:654] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:43.453356 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bb5e8cd427a4fdf9a1ccbc0fb4caaf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.454495 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ca34b2e67ca14f05822bd6615eb5d107"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.455075 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca34b2e67ca14f05822bd6615eb5d107 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.456305 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9245aa65a3734982b3e84187cc176312"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.456763 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9245aa65a3734982b3e84187cc176312 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.457839 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a43790ad2a5d42c986031de7a39591bf" } new_table_name: "default.Table_c19ff63b162e4df9b84be3f39a8ae63f" modify_external_catalogs: true
I20250629 01:51:43.458084 1155 tablet_bootstrap.cc:492] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.458180 1155 ts_tablet_manager.cc:1397] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20250629 01:51:43.458787 1155 raft_consensus.cc:357] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.458914 1155 raft_consensus.cc:383] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.458983 1155 raft_consensus.cc:738] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.459678 1155 consensus_queue.cc:260] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.460180 1155 ts_tablet_manager.cc:1428] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:43.460392 1155 tablet_bootstrap.cc:492] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.462077 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "990d1fcdac1c406facb2da722421d7f4" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:43.462240 777 raft_consensus.cc:2466] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:43.462474 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "990d1fcdac1c406facb2da722421d7f4" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:43.462622 775 raft_consensus.cc:3058] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.464356 1050 raft_consensus.cc:2466] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:43.465032 775 raft_consensus.cc:2466] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:43.466305 986 leader_election.cc:304] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.468410 1155 tablet_bootstrap.cc:654] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.472303 1156 ts_tablet_manager.cc:1918] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:43.472445 1156 log.cc:1199] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a229b4f9d19147a5990f799d1dc0e5d7
I20250629 01:51:43.472939 1156 ts_tablet_manager.cc:1939] T a229b4f9d19147a5990f799d1dc0e5d7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:43.476797 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4080248559414e1b92597c0367d2530c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.477209 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4080248559414e1b92597c0367d2530c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.477921 1157 ts_tablet_manager.cc:1918] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.478057 1157 log.cc:1199] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a229b4f9d19147a5990f799d1dc0e5d7
I20250629 01:51:43.478396 1157 ts_tablet_manager.cc:1939] T a229b4f9d19147a5990f799d1dc0e5d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.479061 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a229b4f9d19147a5990f799d1dc0e5d7 (table default.Table_654038c5004a437482989db6feea6573 [id=0c84d077516244288b296db0048f3497]) successfully deleted
I20250629 01:51:43.479013 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c1cb87a8cbbd4ab5b62e65e5c31e2c95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.479260 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a485e0e6e2f04aa6bd67a62f6d30be26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a485e0e6e2f04aa6bd67a62f6d30be26 already in progress: creating tablet
I20250629 01:51:43.479377 1157 tablet_replica.cc:331] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:43.479422 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1cb87a8cbbd4ab5b62e65e5c31e2c95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:43.479449 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0d925ffce89b40258b29ec021bfa5f92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d925ffce89b40258b29ec021bfa5f92 already in progress: creating tablet
I20250629 01:51:43.479585 1157 raft_consensus.cc:2241] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:43.479651 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0a574f95ac2a4193a563e63cbe120768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a574f95ac2a4193a563e63cbe120768 already in progress: creating tablet
I20250629 01:51:43.479776 1157 raft_consensus.cc:2270] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:43.479806 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9cf553b291854a6f9e7ed78b5db5f18c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf553b291854a6f9e7ed78b5db5f18c already in progress: creating tablet
I20250629 01:51:43.479959 1159 tablet_bootstrap.cc:492] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
W20250629 01:51:43.479987 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e1fe3880e51345269e852014d0d60bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1fe3880e51345269e852014d0d60bb6 already in progress: creating tablet
I20250629 01:51:43.480113 1159 ts_tablet_manager.cc:1397] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.056s user 0.015s sys 0.035s
I20250629 01:51:43.480674 755 tablet_service.cc:1468] Processing CreateTablet for tablet ef976703d61442f8987ff32070a33592 (DEFAULT_TABLE table=default.Table_f220872b43b3433694c2f522f6e54acd [id=b1662df1e42f43eeae59f02a3e446006]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.480984 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef976703d61442f8987ff32070a33592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.480943 1159 raft_consensus.cc:357] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.481128 1159 raft_consensus.cc:383] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.481212 1159 raft_consensus.cc:738] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.481411 1159 consensus_queue.cc:260] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.482007 1159 ts_tablet_manager.cc:1428] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:43.482295 1159 tablet_bootstrap.cc:492] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.482522 1158 ts_tablet_manager.cc:1918] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:43.482650 1158 log.cc:1199] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d1956051272e450ab2ba3322ba23ab69
I20250629 01:51:43.483021 1158 ts_tablet_manager.cc:1939] T d1956051272e450ab2ba3322ba23ab69 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:43.484477 1157 ts_tablet_manager.cc:1905] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.485654 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
I20250629 01:51:43.485818 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:43.485958 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_f1a2e572d6ab48b8a6f6a2b818324c38 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 1 got 0
I20250629 01:51:43.487861 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ef976703d61442f8987ff32070a33592 (DEFAULT_TABLE table=default.Table_f220872b43b3433694c2f522f6e54acd [id=b1662df1e42f43eeae59f02a3e446006]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.488224 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef976703d61442f8987ff32070a33592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.489656 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 1bb4ff3220cf4d25b680235f1b4882ed (DEFAULT_TABLE table=default.Table_084d0d93b035470e8e5dbd670d05c9e3 [id=5aec29cfb498423daa8d43eaf2157d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.489957 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb4ff3220cf4d25b680235f1b4882ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.492354 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_afb98d5207ae4d2d8b97b8577ae3a157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.492734 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afb98d5207ae4d2d8b97b8577ae3a157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.497591 1013 tablet_service.cc:1468] Processing CreateTablet for tablet d159e69137d34899ace5c245638a3c0f (DEFAULT_TABLE table=default.Table_165130da0f884c2f99cc4de1d8689cd1 [id=ffcf2d1691464880b592ca1451808556]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.497843 745 tablet_service.cc:1468] Processing CreateTablet for tablet 56b9ef027c9d42048e4a6dff221df793 (DEFAULT_TABLE table=default.Table_0072df8d087f4e8e9541d7f90a27bf4b [id=24d959060a484a2bab405fa65b92abf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.497932 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d159e69137d34899ace5c245638a3c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.498157 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b9ef027c9d42048e4a6dff221df793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.498883 757 tablet_service.cc:1468] Processing CreateTablet for tablet 1bb4ff3220cf4d25b680235f1b4882ed (DEFAULT_TABLE table=default.Table_084d0d93b035470e8e5dbd670d05c9e3 [id=5aec29cfb498423daa8d43eaf2157d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.499148 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb4ff3220cf4d25b680235f1b4882ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.497596 1017 tablet_service.cc:1468] Processing CreateTablet for tablet f5941dd6381041538be6c92858d6ba13 (DEFAULT_TABLE table=default.Table_177db27d5e8b4755ad499157679a6f95 [id=0fb1390e9626454eba8e8d5c634e3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.499775 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5941dd6381041538be6c92858d6ba13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.501135 1159 tablet_bootstrap.cc:654] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.501700 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec (DEFAULT_TABLE table=default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.501979 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6f3f19a4d441f58c8c6099e04f82ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.503156 1157 ts_tablet_manager.cc:1918] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.503252 1157 log.cc:1199] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/64477468c6da4b4ba282d8530eb5cbb5
I20250629 01:51:43.503535 1157 ts_tablet_manager.cc:1939] T 64477468c6da4b4ba282d8530eb5cbb5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.504413 754 tablet_service.cc:1468] Processing CreateTablet for tablet f5941dd6381041538be6c92858d6ba13 (DEFAULT_TABLE table=default.Table_177db27d5e8b4755ad499157679a6f95 [id=0fb1390e9626454eba8e8d5c634e3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.504721 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5941dd6381041538be6c92858d6ba13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.504844 1157 tablet_replica.cc:331] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.505003 1157 raft_consensus.cc:2241] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.505128 1157 raft_consensus.cc:2270] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.505342 747 tablet_service.cc:1468] Processing CreateTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb (DEFAULT_TABLE table=default.Table_c61ad1c9857141faad549e9cc863671e [id=16fca49978904e5c880caa5d56572c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.505617 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d79ac3c5f04b5db295c93c3aae92cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.506655 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb (DEFAULT_TABLE table=default.Table_c61ad1c9857141faad549e9cc863671e [id=16fca49978904e5c880caa5d56572c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.506672 750 tablet_service.cc:1468] Processing CreateTablet for tablet d159e69137d34899ace5c245638a3c0f (DEFAULT_TABLE table=default.Table_165130da0f884c2f99cc4de1d8689cd1 [id=ffcf2d1691464880b592ca1451808556]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.506929 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d159e69137d34899ace5c245638a3c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.506937 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d79ac3c5f04b5db295c93c3aae92cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.509497 1157 ts_tablet_manager.cc:1905] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.515308 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 56b9ef027c9d42048e4a6dff221df793 (DEFAULT_TABLE table=default.Table_0072df8d087f4e8e9541d7f90a27bf4b [id=24d959060a484a2bab405fa65b92abf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.515599 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b9ef027c9d42048e4a6dff221df793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.516413 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:43.522071 884 tablet_service.cc:1468] Processing CreateTablet for tablet 56b9ef027c9d42048e4a6dff221df793 (DEFAULT_TABLE table=default.Table_0072df8d087f4e8e9541d7f90a27bf4b [id=24d959060a484a2bab405fa65b92abf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.522325 891 tablet_service.cc:1468] Processing CreateTablet for tablet ef976703d61442f8987ff32070a33592 (DEFAULT_TABLE table=default.Table_f220872b43b3433694c2f522f6e54acd [id=b1662df1e42f43eeae59f02a3e446006]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.522488 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b9ef027c9d42048e4a6dff221df793. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.522729 1430 raft_consensus.cc:491] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.522863 1430 raft_consensus.cc:513] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.524029 1430 leader_election.cc:290] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.525326 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec (DEFAULT_TABLE table=default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.525601 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6f3f19a4d441f58c8c6099e04f82ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.526175 885 tablet_service.cc:1468] Processing CreateTablet for tablet 1bb4ff3220cf4d25b680235f1b4882ed (DEFAULT_TABLE table=default.Table_084d0d93b035470e8e5dbd670d05c9e3 [id=5aec29cfb498423daa8d43eaf2157d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.526492 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bb4ff3220cf4d25b680235f1b4882ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.526669 1157 ts_tablet_manager.cc:1918] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.526804 1157 log.cc:1199] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/dd6013bb40c248f29f40eab2c3008f6b
I20250629 01:51:43.527148 1157 ts_tablet_manager.cc:1939] T dd6013bb40c248f29f40eab2c3008f6b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.506254 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec (DEFAULT_TABLE table=default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.528422 850 leader_election.cc:304] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:43.528498 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c6f3f19a4d441f58c8c6099e04f82ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.528615 850 raft_consensus.cc:2684] T c5a1e71ab9a84caea5c917d422336bc8 P 443c4bd3c5d548ee8cd347d2b4de5293: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:43.529292 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef976703d61442f8987ff32070a33592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.532393 887 tablet_service.cc:1468] Processing CreateTablet for tablet f5941dd6381041538be6c92858d6ba13 (DEFAULT_TABLE table=default.Table_177db27d5e8b4755ad499157679a6f95 [id=0fb1390e9626454eba8e8d5c634e3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.532477 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a229b4f9d19147a5990f799d1dc0e5d7 (table default.Table_654038c5004a437482989db6feea6573 [id=0c84d077516244288b296db0048f3497]) successfully deleted
I20250629 01:51:43.532660 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 64477468c6da4b4ba282d8530eb5cbb5 (table default.Table_6f9bb7305f9d42fbb5b12690468aed5f [id=81e46758eee54c7caae081f9232e9071]) successfully deleted
I20250629 01:51:43.532783 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5941dd6381041538be6c92858d6ba13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.532922 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "990d1fcdac1c406facb2da722421d7f4" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:43.533118 914 raft_consensus.cc:3058] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.535580 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet dd6013bb40c248f29f40eab2c3008f6b (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:51:43.535794 1430 raft_consensus.cc:491] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.535933 1430 raft_consensus.cc:513] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.536319 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4701123dfa467ebb6c95f9742b3874" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:43.536509 775 raft_consensus.cc:2466] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:43.536473 888 tablet_service.cc:1468] Processing CreateTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb (DEFAULT_TABLE table=default.Table_c61ad1c9857141faad549e9cc863671e [id=16fca49978904e5c880caa5d56572c22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.536856 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d79ac3c5f04b5db295c93c3aae92cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.537191 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e1fe3880e51345269e852014d0d60bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1fe3880e51345269e852014d0d60bb6 already in progress: creating tablet
I20250629 01:51:43.538338 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:43.538491 775 raft_consensus.cc:2466] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:43.538618 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:43.539228 850 leader_election.cc:304] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.539436 1425 raft_consensus.cc:2802] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.539480 850 leader_election.cc:304] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.539543 1425 raft_consensus.cc:491] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.539619 1425 raft_consensus.cc:3058] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.540472 894 tablet_service.cc:1468] Processing CreateTablet for tablet d159e69137d34899ace5c245638a3c0f (DEFAULT_TABLE table=default.Table_165130da0f884c2f99cc4de1d8689cd1 [id=ffcf2d1691464880b592ca1451808556]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.541029 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d159e69137d34899ace5c245638a3c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.536474 1430 leader_election.cc:290] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.545228 1364 raft_consensus.cc:2802] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:43.545696 1364 raft_consensus.cc:695] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:43.546000 1364 consensus_queue.cc:237] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.553812 1155 tablet_bootstrap.cc:492] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.553948 1155 ts_tablet_manager.cc:1397] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.004s
I20250629 01:51:43.554554 1155 raft_consensus.cc:357] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.554858 1155 raft_consensus.cc:383] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.554942 1155 raft_consensus.cc:738] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.555151 1155 consensus_queue.cc:260] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.555191 1425 raft_consensus.cc:513] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.556378 1425 leader_election.cc:290] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.557093 1155 ts_tablet_manager.cc:1428] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:43.557303 914 raft_consensus.cc:2466] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:43.557397 1155 tablet_bootstrap.cc:492] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.560976 1155 tablet_bootstrap.cc:654] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.569453 1155 tablet_bootstrap.cc:492] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.569727 1155 ts_tablet_manager.cc:1397] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20250629 01:51:43.570713 1155 raft_consensus.cc:357] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.570946 1155 raft_consensus.cc:383] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.571058 1155 raft_consensus.cc:738] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.571313 1155 consensus_queue.cc:260] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.571975 1155 ts_tablet_manager.cc:1428] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:43.572360 1155 tablet_bootstrap.cc:492] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.575122 1155 tablet_bootstrap.cc:654] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.589566 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 35990d206ba64159ab05ab7b7b19d064
W20250629 01:51:43.589601 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 50149us
I20250629 01:51:43.590344 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4701123dfa467ebb6c95f9742b3874" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:43.590513 775 raft_consensus.cc:3058] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.591279 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8799b1858bf2408a85f0956578639393
I20250629 01:51:43.591820 447 master_service.cc:948] ReplaceTablet: received request to replace tablet fe14d1d54db74a99b05cf8b1a60827a9 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.592932 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:43.593291 775 raft_consensus.cc:2466] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:43.593358 454 catalog_manager.cc:6652] ReplaceTablet: tablet f8b38c20eb1849b1a9f8acd40315eef5 deleted and replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec
I20250629 01:51:43.593603 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5ce4c958f3d4456fb56450ca4cc26221" } modify_external_catalogs: true
I20250629 01:51:43.593776 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5ce4c958f3d4456fb56450ca4cc26221" } modify_external_catalogs: true
I20250629 01:51:43.594942 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_05483b691b564e7482b9d04d0afb1b29 [id=6a41ce2e3dcf43ce9edb6392a244d9a2] alter complete (version 3)
I20250629 01:51:43.595880 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6b6adcddb17145a7a5a51e2f83ada474" } new_table_name: "default.Table_94c8eccb1a094394bad377478edb243d" modify_external_catalogs: true
I20250629 01:51:43.597582 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4701123dfa467ebb6c95f9742b3874" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:43.598067 1427 raft_consensus.cc:2802] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.598192 1427 raft_consensus.cc:491] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.598290 1427 raft_consensus.cc:3058] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:43.599335 850 leader_election.cc:343] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.601238 1427 raft_consensus.cc:513] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.601804 1427 leader_election.cc:290] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:43.603634 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e508393b9cb044c4b90194c8aab3184a consensus state has no leader
I20250629 01:51:43.604015 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8bb871e6aa1745a6a3007e6b15ee75cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.604465 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bb871e6aa1745a6a3007e6b15ee75cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.605825 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 575006c12f7c41658e234f7c6c425d69 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.606761 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ed143377709465d9ae5df7c2ec3ca47 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.607237 446 master_service.cc:948] ReplaceTablet: received request to replace tablet b6688e9665f24e73b80802d47a80a2cd from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:43.607589 850 leader_election.cc:343] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.608070 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:43.608261 775 raft_consensus.cc:3058] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.611070 775 raft_consensus.cc:2466] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:43.611420 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_60c45c3633104e8aa039d5d92633c213" } new_table_name: "default.Table_59f54bc37cc14d9fbccad27fdbc6e62e" modify_external_catalogs: true
I20250629 01:51:43.613216 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:43.613673 850 leader_election.cc:304] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:43.613935 1425 raft_consensus.cc:2802] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:43.614066 1425 raft_consensus.cc:695] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:43.614338 1425 consensus_queue.cc:237] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.614064 850 leader_election.cc:304] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.617683 1430 raft_consensus.cc:2802] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:43.617789 1430 raft_consensus.cc:695] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:43.618054 1430 consensus_queue.cc:237] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.618414 1465 raft_consensus.cc:491] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.618571 1465 raft_consensus.cc:513] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.619139 1465 leader_election.cc:290] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.620008 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet fe14d1d54db74a99b05cf8b1a60827a9
I20250629 01:51:43.620501 447 catalog_manager.cc:6652] ReplaceTablet: tablet fe14d1d54db74a99b05cf8b1a60827a9 deleted and replaced by tablet 1c995ce14c9a4966b842fcba520cf87d
I20250629 01:51:43.620849 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f81f54577f464e6bbe68673617d0e0cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.621417 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f81f54577f464e6bbe68673617d0e0cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.622282 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5b835ff8624ebf8956eeb965ce5950" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:43.622751 712 leader_election.cc:343] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.624473 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5b835ff8624ebf8956eeb965ce5950" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:43.625510 710 leader_election.cc:343] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.625664 710 leader_election.cc:304] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:43.626103 1465 raft_consensus.cc:2747] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:43.627786 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_177db27d5e8b4755ad499157679a6f95" } new_table_name: "default.Table_36109461cd234772b6d33074e7fd5a1a" modify_external_catalogs: true
W20250629 01:51:43.627888 850 leader_election.cc:343] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.628386 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4701123dfa467ebb6c95f9742b3874" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:43.629052 1350 raft_consensus.cc:491] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.629114 884 tablet_service.cc:1515] Processing DeleteTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 with delete_type TABLET_DATA_DELETED (replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.629204 1350 raft_consensus.cc:513] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.629427 894 tablet_service.cc:1515] Processing DeleteTablet for tablet b939a8568ac14c339258af925fcb809b with delete_type TABLET_DATA_DELETED (replaced by tablet 34172d05ae40417494cd8f67f4cb981e) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.629693 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 582e063f97ab415cab126845789e6e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.629914 886 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.629922 1350 leader_election.cc:290] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.630465 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 35990d206ba64159ab05ab7b7b19d064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.630668 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3a91740f16744439f55077c4fbadee1" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:43.631356 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_11a668da1f204d02b70b4ccd24b59bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.632017 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11a668da1f204d02b70b4ccd24b59bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.633193 1154 tablet_bootstrap.cc:492] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:43.633337 1154 ts_tablet_manager.cc:1397] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.626s user 0.005s sys 0.008s
I20250629 01:51:43.634048 1154 raft_consensus.cc:357] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.634759 1154 raft_consensus.cc:383] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.634850 1154 raft_consensus.cc:738] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.635051 1154 consensus_queue.cc:260] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.636471 1157 tablet_replica.cc:331] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.636715 1157 raft_consensus.cc:2241] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.636929 1157 raft_consensus.cc:2270] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.637576 1157 ts_tablet_manager.cc:1905] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.618388 448 catalog_manager.cc:5582] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:43.640316 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_f1a2e572d6ab48b8a6f6a2b818324c38 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 1 got 0
I20250629 01:51:43.642234 1158 tablet_replica.cc:331] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:43.642432 1158 raft_consensus.cc:2241] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.645227 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d3a91740f16744439f55077c4fbadee1" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:43.646065 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f8b38c20eb1849b1a9f8acd40315eef5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8b38c20eb1849b1a9f8acd40315eef5 already in progress: creating tablet
I20250629 01:51:43.646159 1157 ts_tablet_manager.cc:1918] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.646288 1157 log.cc:1199] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b939a8568ac14c339258af925fcb809b
I20250629 01:51:43.646756 1157 ts_tablet_manager.cc:1939] T b939a8568ac14c339258af925fcb809b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:43.646807 850 leader_election.cc:343] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:43.647308 850 leader_election.cc:343] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:43.647578 850 leader_election.cc:343] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.647732 1157 tablet_replica.cc:331] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.647961 1157 raft_consensus.cc:2241] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.650362 1158 raft_consensus.cc:2270] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.650907 1158 ts_tablet_manager.cc:1905] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.647758 850 leader_election.cc:304] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:43.652532 1427 raft_consensus.cc:2747] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:43.652710 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b939a8568ac14c339258af925fcb809b (table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]) successfully deleted
I20250629 01:51:43.657188 1157 raft_consensus.cc:2270] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.658021 1157 ts_tablet_manager.cc:1905] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.660207 1158 ts_tablet_manager.cc:1918] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.664160 1158 log.cc:1199] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/7ae96f2da36744198e3d38cb07bb47a0
I20250629 01:51:43.664573 1158 ts_tablet_manager.cc:1939] T 7ae96f2da36744198e3d38cb07bb47a0 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:43.665591 1158 tablet_replica.cc:331] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:43.665819 1158 raft_consensus.cc:2241] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.666026 1158 raft_consensus.cc:2270] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.666584 1158 ts_tablet_manager.cc:1905] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.660751 1154 ts_tablet_manager.cc:1428] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250629 01:51:43.667091 1157 ts_tablet_manager.cc:1918] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.667505 1154 tablet_bootstrap.cc:492] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:43.669346 1154 tablet_bootstrap.cc:654] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.671319 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 35990d206ba64159ab05ab7b7b19d064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.674280 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 35990d206ba64159ab05ab7b7b19d064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.675000 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 8799b1858bf2408a85f0956578639393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.675675 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8799b1858bf2408a85f0956578639393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.676213 745 tablet_service.cc:1515] Processing DeleteTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 with delete_type TABLET_DATA_DELETED (replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.667246 1157 log.cc:1199] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/582e063f97ab415cab126845789e6e20
I20250629 01:51:43.676913 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 with delete_type TABLET_DATA_DELETED (replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.676980 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 75fd4ffcf896419b81a1c0eb7c5aeeec
I20250629 01:51:43.677253 1157 ts_tablet_manager.cc:1939] T 582e063f97ab415cab126845789e6e20 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.677480 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.678033 745 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.678354 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d1956051272e450ab2ba3322ba23ab69 (table default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]) successfully deleted
I20250629 01:51:43.678961 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2c04531a50c54791a64d6537b670d54f" } modify_external_catalogs: true
I20250629 01:51:43.679127 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2c04531a50c54791a64d6537b670d54f" } modify_external_catalogs: true
W20250629 01:51:43.680059 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 829c64e2c71347f4af2c8870cdd8af31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829c64e2c71347f4af2c8870cdd8af31 already in progress: creating tablet
I20250629 01:51:43.680665 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6ada0762e0224924acf20a4507ad2fa6" } modify_external_catalogs: true
I20250629 01:51:43.680822 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6ada0762e0224924acf20a4507ad2fa6" } modify_external_catalogs: true
I20250629 01:51:43.682139 1159 tablet_bootstrap.cc:492] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.682358 1159 ts_tablet_manager.cc:1397] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.200s user 0.036s sys 0.049s
I20250629 01:51:43.683161 1159 raft_consensus.cc:357] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.683351 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 582e063f97ab415cab126845789e6e20 (table default.Table_30a1da646c914cde85078e3e1270a01e [id=26e8b18cac084e4c9e603950a6595c35]) successfully deleted
W20250629 01:51:43.683573 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet fe14d1d54db74a99b05cf8b1a60827a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe14d1d54db74a99b05cf8b1a60827a9 already in progress: creating tablet
I20250629 01:51:43.683820 1159 raft_consensus.cc:383] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.683902 1159 raft_consensus.cc:738] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.684144 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.684139 1159 consensus_queue.cc:260] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.684514 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d2344e007e2f4266aa706d32b9465347"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.684589 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 35990d206ba64159ab05ab7b7b19d064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35990d206ba64159ab05ab7b7b19d064 already in progress: creating tablet
I20250629 01:51:43.684723 1159 ts_tablet_manager.cc:1428] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250629 01:51:43.685083 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2344e007e2f4266aa706d32b9465347 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.685261 1159 tablet_bootstrap.cc:492] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:43.685655 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75fd4ffcf896419b81a1c0eb7c5aeeec already in progress: creating tablet
I20250629 01:51:43.686147 1427 raft_consensus.cc:491] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.686305 1427 raft_consensus.cc:513] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.686937 1427 leader_election.cc:290] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:43.687073 1159 tablet_bootstrap.cc:654] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.687424 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cade2a7984a3413a9690af48b8c28e84" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:43.687593 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 798516e55ca747fcb2942026adb78af0 consensus state has no leader
I20250629 01:51:43.687626 775 raft_consensus.cc:2466] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:43.687907 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c6543e800e6f41d4ac45569c0650a8f4" } new_table_name: "default.Table_55354f7b2e844ba292e478667d1f0d76" modify_external_catalogs: true
I20250629 01:51:43.688185 850 leader_election.cc:304] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:43.688553 1427 raft_consensus.cc:2802] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:43.688663 1427 raft_consensus.cc:491] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:43.688791 1427 raft_consensus.cc:3058] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.690462 1154 tablet_bootstrap.cc:492] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:51:43.690845 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5941dd6381041538be6c92858d6ba13 consensus state has no leader
I20250629 01:51:43.692382 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cade2a7984a3413a9690af48b8c28e84" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:43.692966 850 leader_election.cc:343] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.693543 1427 raft_consensus.cc:513] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.690598 1154 ts_tablet_manager.cc:1397] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250629 01:51:43.694478 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cade2a7984a3413a9690af48b8c28e84" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:43.694681 775 raft_consensus.cc:3058] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:43.694944 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cade2a7984a3413a9690af48b8c28e84" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
W20250629 01:51:43.695467 850 leader_election.cc:343] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:43.695503 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d1d0b27c4dae4594a84e9859774c89d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.695756 1427 leader_election.cc:290] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:43.695991 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1d0b27c4dae4594a84e9859774c89d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.697444 775 raft_consensus.cc:2466] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:43.698020 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 7ae96f2da36744198e3d38cb07bb47a0 (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:43.697990 850 leader_election.cc:304] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:43.698529 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b6688e9665f24e73b80802d47a80a2cd
I20250629 01:51:43.699121 446 catalog_manager.cc:6652] ReplaceTablet: tablet b6688e9665f24e73b80802d47a80a2cd deleted and replaced by tablet 52be184d0480495a894442d6f72a0f12
I20250629 01:51:43.699222 1159 tablet_bootstrap.cc:492] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.699373 1159 ts_tablet_manager.cc:1397] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20250629 01:51:43.699422 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ed13689efff54739bc1cd029699b4384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.699877 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed13689efff54739bc1cd029699b4384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.700363 1159 raft_consensus.cc:357] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.700615 1159 raft_consensus.cc:383] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.700695 1159 raft_consensus.cc:738] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.700865 1159 consensus_queue.cc:260] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.701022 1154 raft_consensus.cc:357] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.701169 1427 raft_consensus.cc:2802] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:43.701189 1154 raft_consensus.cc:383] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.701316 1427 raft_consensus.cc:695] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:43.701344 1159 ts_tablet_manager.cc:1428] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:43.701452 1154 raft_consensus.cc:738] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.701603 1159 tablet_bootstrap.cc:492] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.701692 1154 consensus_queue.cc:260] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.702164 891 tablet_service.cc:1515] Processing DeleteTablet for tablet b6688e9665f24e73b80802d47a80a2cd with delete_type TABLET_DATA_DELETED (replaced by tablet 52be184d0480495a894442d6f72a0f12) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.702551 1154 ts_tablet_manager.cc:1428] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250629 01:51:43.702885 1154 tablet_bootstrap.cc:492] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:43.701545 1427 consensus_queue.cc:237] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.704845 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7fec8ba2ba174536baceb16beaa50ab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.705973 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fec8ba2ba174536baceb16beaa50ab8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.707638 1154 tablet_bootstrap.cc:654] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.708926 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.709566 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.670506 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 8799b1858bf2408a85f0956578639393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.711782 1157 tablet_replica.cc:331] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.711994 1157 raft_consensus.cc:2241] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.712250 1157 raft_consensus.cc:2270] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.712950 1157 ts_tablet_manager.cc:1905] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.716965 1159 tablet_bootstrap.cc:654] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:43.717028 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 35990d206ba64159ab05ab7b7b19d064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35990d206ba64159ab05ab7b7b19d064 already in progress: creating tablet
W20250629 01:51:43.717278 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8799b1858bf2408a85f0956578639393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8799b1858bf2408a85f0956578639393 already in progress: creating tablet
W20250629 01:51:43.718096 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f8b38c20eb1849b1a9f8acd40315eef5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8b38c20eb1849b1a9f8acd40315eef5 already in progress: creating tablet
W20250629 01:51:43.718292 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet fe14d1d54db74a99b05cf8b1a60827a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe14d1d54db74a99b05cf8b1a60827a9 already in progress: creating tablet
W20250629 01:51:43.718520 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75fd4ffcf896419b81a1c0eb7c5aeeec already in progress: creating tablet
I20250629 01:51:43.718968 745 tablet_service.cc:1515] Processing DeleteTablet for tablet b6688e9665f24e73b80802d47a80a2cd with delete_type TABLET_DATA_DELETED (replaced by tablet 52be184d0480495a894442d6f72a0f12) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.719219 1156 tablet_replica.cc:331] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:43.719444 1156 raft_consensus.cc:2241] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:43.719813 1156 raft_consensus.cc:2270] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.720485 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b6688e9665f24e73b80802d47a80a2cd with delete_type TABLET_DATA_DELETED (replaced by tablet 52be184d0480495a894442d6f72a0f12) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.720535 1156 ts_tablet_manager.cc:1905] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.726809 1156 ts_tablet_manager.cc:1918] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:43.726922 1156 log.cc:1199] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b6688e9665f24e73b80802d47a80a2cd
I20250629 01:51:43.727301 1156 ts_tablet_manager.cc:1939] T b6688e9665f24e73b80802d47a80a2cd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:43.727634 1158 ts_tablet_manager.cc:1918] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.728209 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]) successfully deleted
I20250629 01:51:43.729238 1157 ts_tablet_manager.cc:1918] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:43.729367 1157 log.cc:1199] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b6688e9665f24e73b80802d47a80a2cd
I20250629 01:51:43.729502 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet af9ab324ce7c488cbaf6e0f0322b5c58
I20250629 01:51:43.729801 1157 ts_tablet_manager.cc:1939] T b6688e9665f24e73b80802d47a80a2cd P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.732509 1159 tablet_bootstrap.cc:492] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.732661 1159 ts_tablet_manager.cc:1397] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250629 01:51:43.732807 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]) successfully deleted
I20250629 01:51:43.727759 1158 log.cc:1199] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/c5a1e71ab9a84caea5c917d422336bc8
I20250629 01:51:43.733446 1159 raft_consensus.cc:357] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.733551 1158 ts_tablet_manager.cc:1939] T c5a1e71ab9a84caea5c917d422336bc8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:43.733600 1159 raft_consensus.cc:383] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.733649 1159 raft_consensus.cc:738] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.733937 1159 consensus_queue.cc:260] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.734390 1159 ts_tablet_manager.cc:1428] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:43.734686 1159 tablet_bootstrap.cc:492] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.738608 1159 tablet_bootstrap.cc:654] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:43.738956 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8799b1858bf2408a85f0956578639393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8799b1858bf2408a85f0956578639393 already in progress: creating tablet
I20250629 01:51:43.740144 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 257) took 1046 ms. Trace:
I20250629 01:51:43.740249 1158 rpcz_store.cc:276] 0629 01:51:42.689185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:42.689524 (+ 339us) service_pool.cc:225] Handling call
0629 01:51:43.665528 (+976004us) ts_tablet_manager.cc:1145] Deleting tablet c5a1e71ab9a84caea5c917d422336bc8
0629 01:51:43.665536 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:43.704645 (+ 39109us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:43.727534 (+ 22889us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:43.735545 (+ 8011us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":823290}
I20250629 01:51:43.742614 1158 tablet_replica.cc:331] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:43.742772 1158 raft_consensus.cc:2241] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.742937 1158 raft_consensus.cc:2270] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.742995 888 tablet_service.cc:1515] Processing DeleteTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.743396 1158 ts_tablet_manager.cc:1905] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:43.743481 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet af9ab324ce7c488cbaf6e0f0322b5c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af9ab324ce7c488cbaf6e0f0322b5c58 already in progress: creating tablet
I20250629 01:51:43.744570 1159 tablet_bootstrap.cc:492] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.744681 1159 ts_tablet_manager.cc:1397] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250629 01:51:43.745347 1159 raft_consensus.cc:357] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.745489 1159 raft_consensus.cc:383] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.745563 1159 raft_consensus.cc:738] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.745787 1159 consensus_queue.cc:260] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.746199 1159 ts_tablet_manager.cc:1428] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:43.746439 1159 tablet_bootstrap.cc:492] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.751448 1159 tablet_bootstrap.cc:654] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.751720 745 tablet_service.cc:1515] Processing DeleteTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.752116 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.752626 1382 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250629 01:51:43.753710 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 consensus state has no leader
I20250629 01:51:43.755504 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet c5a1e71ab9a84caea5c917d422336bc8 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
W20250629 01:51:43.755784 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet af9ab324ce7c488cbaf6e0f0322b5c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af9ab324ce7c488cbaf6e0f0322b5c58 already in progress: creating tablet
I20250629 01:51:43.755745 1387 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:43.758769 1155 tablet_bootstrap.cc:492] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.758934 1155 ts_tablet_manager.cc:1397] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.187s user 0.012s sys 0.044s
I20250629 01:51:43.760300 1155 raft_consensus.cc:357] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.760515 1155 raft_consensus.cc:383] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.760591 1155 raft_consensus.cc:738] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.760874 1155 consensus_queue.cc:260] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.761397 1155 ts_tablet_manager.cc:1428] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:43.761745 1155 tablet_bootstrap.cc:492] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.764145 1155 tablet_bootstrap.cc:654] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.771922 1155 tablet_bootstrap.cc:492] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.772079 1155 ts_tablet_manager.cc:1397] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20250629 01:51:43.772971 1155 raft_consensus.cc:357] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.773306 1155 raft_consensus.cc:383] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.773411 1155 raft_consensus.cc:738] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.773699 1155 consensus_queue.cc:260] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.774273 1155 ts_tablet_manager.cc:1428] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:43.774578 1155 tablet_bootstrap.cc:492] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.776470 1155 tablet_bootstrap.cc:654] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.778648 1155 tablet_bootstrap.cc:492] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:43.778815 1155 ts_tablet_manager.cc:1397] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250629 01:51:43.779614 1155 raft_consensus.cc:357] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.779847 1155 raft_consensus.cc:383] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.779930 1155 raft_consensus.cc:738] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.780231 1155 consensus_queue.cc:260] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.780835 1155 ts_tablet_manager.cc:1428] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:43.781157 1155 tablet_bootstrap.cc:492] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:43.785647 1155 tablet_bootstrap.cc:654] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.787513 745 tablet_service.cc:1468] Processing CreateTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d (DEFAULT_TABLE table=default.Table_ca34b2e67ca14f05822bd6615eb5d107 [id=c03cb89aed104caa852122ebd5722929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.787360 454 catalog_manager.cc:5582] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:43.787763 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
I20250629 01:51:43.787889 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cabb0298771e4fd5be29e530d3a8ed3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.787985 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:43.788251 1154 tablet_bootstrap.cc:492] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:43.788426 1154 ts_tablet_manager.cc:1397] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.002s
I20250629 01:51:43.789830 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_f1a2e572d6ab48b8a6f6a2b818324c38 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 1 got 0
I20250629 01:51:43.790024 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]). Expected version 1 got 0
I20250629 01:51:43.790182 1154 raft_consensus.cc:357] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.790401 1154 raft_consensus.cc:383] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.790473 1154 raft_consensus.cc:738] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.790743 1154 consensus_queue.cc:260] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.791317 1154 ts_tablet_manager.cc:1428] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:51:43.791378 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]). Expected version 1 got 0
I20250629 01:51:43.791666 1154 tablet_bootstrap.cc:492] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:43.794515 888 tablet_service.cc:1468] Processing CreateTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d (DEFAULT_TABLE table=default.Table_ca34b2e67ca14f05822bd6615eb5d107 [id=c03cb89aed104caa852122ebd5722929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.794940 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cabb0298771e4fd5be29e530d3a8ed3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.796479 1339 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:43.803421 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9bbbeb43473a4f6f892056a30f62589f (DEFAULT_TABLE table=default.Table_9245aa65a3734982b3e84187cc176312 [id=8705c82c65df415abf3299b10b90f48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.803875 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbbeb43473a4f6f892056a30f62589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.805236 1154 tablet_bootstrap.cc:654] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.809114 1158 ts_tablet_manager.cc:1918] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.809240 1158 log.cc:1199] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/2d44151acb0d4bf4a630c086b1ac8ff7
I20250629 01:51:43.809600 1158 ts_tablet_manager.cc:1939] T 2d44151acb0d4bf4a630c086b1ac8ff7 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:43.811985 1159 tablet_bootstrap.cc:492] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.812198 1159 ts_tablet_manager.cc:1397] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.019s
I20250629 01:51:43.812808 1154 tablet_bootstrap.cc:492] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:43.812943 1154 ts_tablet_manager.cc:1397] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250629 01:51:43.812923 1159 raft_consensus.cc:357] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.813104 1159 raft_consensus.cc:383] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.813179 1159 raft_consensus.cc:738] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.813392 1159 consensus_queue.cc:260] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.813944 1159 ts_tablet_manager.cc:1428] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:43.815265 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4e354fce951943d69e4c666897f5caac (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.815594 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e354fce951943d69e4c666897f5caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.816469 1159 tablet_bootstrap.cc:492] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.818111 1159 tablet_bootstrap.cc:654] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.820530 1154 raft_consensus.cc:357] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.820819 1154 raft_consensus.cc:383] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.820957 1154 raft_consensus.cc:738] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.821224 1154 consensus_queue.cc:260] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.823625 1154 ts_tablet_manager.cc:1428] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250629 01:51:43.823904 1154 tablet_bootstrap.cc:492] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:43.823992 1159 tablet_bootstrap.cc:492] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:43.824136 1159 ts_tablet_manager.cc:1397] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20250629 01:51:43.826344 1159 raft_consensus.cc:357] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.826507 1159 raft_consensus.cc:383] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:43.826623 1159 raft_consensus.cc:738] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:43.832401 887 tablet_service.cc:1468] Processing CreateTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 (DEFAULT_TABLE table=default.Table_6f6d8cf186fc4934ac5690bac0754373 [id=db3232e7e81240a699e87310dc0627d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.832779 1154 tablet_bootstrap.cc:654] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:43.832813 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40c1a8eec9549a8bdf695d6516440a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.834324 1158 tablet_replica.cc:331] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:43.834532 1158 raft_consensus.cc:2241] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.834685 1158 raft_consensus.cc:2270] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.834693 895 tablet_service.cc:1468] Processing CreateTablet for tablet ddc24d0e668c49ac825b2578849ab2a3 (DEFAULT_TABLE table=default.Table_adf0dc244a084dcf82f6cb6f22ecc128 [id=d3d515038aa84669a3a64824de88f06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.835034 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc24d0e668c49ac825b2578849ab2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.835196 1158 ts_tablet_manager.cc:1905] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.836370 894 tablet_service.cc:1468] Processing CreateTablet for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (DEFAULT_TABLE table=default.Table_acc52ee4aee240dc82efdc80f09faa91 [id=3d577c959e5b4549a57fa4cdc320a0d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.836721 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.841068 884 tablet_service.cc:1468] Processing CreateTablet for tablet 3b55c66136ba43d7b535750e04b72308 (DEFAULT_TABLE table=default.Table_5bb5e8cd427a4fdf9a1ccbc0fb4caaf4 [id=2ebdb5f48c7b417bb521dd2271922d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.841521 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b55c66136ba43d7b535750e04b72308. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.844341 885 tablet_service.cc:1468] Processing CreateTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b (DEFAULT_TABLE table=default.Table_f128352ab4bd4f258d90d9a4022577a8 [id=b516da4148384818bcc69e4259f20ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.844638 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d21dfd48ea4a95a2fcecde6735974b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.846410 745 tablet_service.cc:1515] Processing DeleteTablet for tablet a229b4f9d19147a5990f799d1dc0e5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.846740 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.847081 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 64477468c6da4b4ba282d8530eb5cbb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:39 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:43.847448 1016 tablet_service.cc:1468] Processing CreateTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d (DEFAULT_TABLE table=default.Table_ca34b2e67ca14f05822bd6615eb5d107 [id=c03cb89aed104caa852122ebd5722929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.847855 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cabb0298771e4fd5be29e530d3a8ed3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.848413 889 tablet_service.cc:1468] Processing CreateTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 (DEFAULT_TABLE table=default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.848820 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3a65c9c1c740b1865b96a6e2aea524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.849422 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (DEFAULT_TABLE table=default.Table_acc52ee4aee240dc82efdc80f09faa91 [id=3d577c959e5b4549a57fa4cdc320a0d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.849767 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.852445 1156 tablet_replica.cc:331] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:43.852519 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4e354fce951943d69e4c666897f5caac (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.852650 1156 raft_consensus.cc:2241] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.852790 1156 raft_consensus.cc:2270] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.852885 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e354fce951943d69e4c666897f5caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.853281 1156 ts_tablet_manager.cc:1905] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.854774 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4e354fce951943d69e4c666897f5caac (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.855114 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e354fce951943d69e4c666897f5caac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.857544 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 3b55c66136ba43d7b535750e04b72308 (DEFAULT_TABLE table=default.Table_5bb5e8cd427a4fdf9a1ccbc0fb4caaf4 [id=2ebdb5f48c7b417bb521dd2271922d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.857944 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b55c66136ba43d7b535750e04b72308. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.858265 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet e508393b9cb044c4b90194c8aab3184a consensus state has no leader
I20250629 01:51:43.859122 757 tablet_service.cc:1468] Processing CreateTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 (DEFAULT_TABLE table=default.Table_6f6d8cf186fc4934ac5690bac0754373 [id=db3232e7e81240a699e87310dc0627d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.859439 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40c1a8eec9549a8bdf695d6516440a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.861050 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 2d44151acb0d4bf4a630c086b1ac8ff7 (table default.Table_d219b9a9062745bca33e3a0fcf553be0 [id=006d3a71c3bc4d1190910bec2c905428]) successfully deleted
W20250629 01:51:43.861253 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 34172d05ae40417494cd8f67f4cb981e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34172d05ae40417494cd8f67f4cb981e already in progress: creating tablet
W20250629 01:51:43.862993 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b7c7f2ac44734a39ab6434a74c91b9e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c7f2ac44734a39ab6434a74c91b9e4 already in progress: creating tablet
I20250629 01:51:43.864954 747 tablet_service.cc:1468] Processing CreateTablet for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (DEFAULT_TABLE table=default.Table_acc52ee4aee240dc82efdc80f09faa91 [id=3d577c959e5b4549a57fa4cdc320a0d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.865314 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.868324 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3b55c66136ba43d7b535750e04b72308 (DEFAULT_TABLE table=default.Table_5bb5e8cd427a4fdf9a1ccbc0fb4caaf4 [id=2ebdb5f48c7b417bb521dd2271922d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.868609 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b55c66136ba43d7b535750e04b72308. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.863350 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ecc7812932cb496d8e92eee3befdce44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.871872 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecc7812932cb496d8e92eee3befdce44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.863425 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3906fb981a8640e1906a78a321c479b2" } modify_external_catalogs: true
I20250629 01:51:43.863719 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4799f303eeb041edb1117ae3caa45b62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.872321 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4799f303eeb041edb1117ae3caa45b62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.864065 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_815cc062f73f433da58e8a2fcc368854"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.872685 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_815cc062f73f433da58e8a2fcc368854 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.865231 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_58a60b60dab64affa9d8b5b1acf53326" } modify_external_catalogs: true
I20250629 01:51:43.873013 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2c253a7938ff45d9aeb8a8d6d9324983" } new_table_name: "default.Table_5f4d9b39ee1546eca5a1fb0701b2b93f" modify_external_catalogs: true
I20250629 01:51:43.873386 750 tablet_service.cc:1468] Processing CreateTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b (DEFAULT_TABLE table=default.Table_f128352ab4bd4f258d90d9a4022577a8 [id=b516da4148384818bcc69e4259f20ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.873698 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d21dfd48ea4a95a2fcecde6735974b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.876317 754 tablet_service.cc:1468] Processing CreateTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 (DEFAULT_TABLE table=default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.876673 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3a65c9c1c740b1865b96a6e2aea524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.879153 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
I20250629 01:51:43.880323 741 tablet_service.cc:1468] Processing CreateTablet for tablet ddc24d0e668c49ac825b2578849ab2a3 (DEFAULT_TABLE table=default.Table_adf0dc244a084dcf82f6cb6f22ecc128 [id=d3d515038aa84669a3a64824de88f06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.880685 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc24d0e668c49ac825b2578849ab2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.884145 888 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.884637 1157 tablet_replica.cc:331] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.885599 1157 raft_consensus.cc:2241] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:43.885848 1157 pending_rounds.cc:70] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Trying to abort 1 pending ops.
I20250629 01:51:43.885936 1157 pending_rounds.cc:77] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759157573779456 op_type: NO_OP noop_request { }
I20250629 01:51:43.886039 1157 raft_consensus.cc:2887] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:43.886031 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9bbbeb43473a4f6f892056a30f62589f (DEFAULT_TABLE table=default.Table_9245aa65a3734982b3e84187cc176312 [id=8705c82c65df415abf3299b10b90f48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.886142 1157 raft_consensus.cc:2270] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.886994 1157 ts_tablet_manager.cc:1905] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.887694 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbbeb43473a4f6f892056a30f62589f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:43.887636 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 81b32bac5a2144d1943285f5f446d53d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81b32bac5a2144d1943285f5f446d53d already in progress: creating tablet
W20250629 01:51:43.891160 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 86832a3c26224c7ea0b9b4b6866d3880 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86832a3c26224c7ea0b9b4b6866d3880 already in progress: creating tablet
I20250629 01:51:43.904709 895 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a91740f16744439f55077c4fbadee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.907042 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
I20250629 01:51:43.907294 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3906fb981a8640e1906a78a321c479b2" } modify_external_catalogs: true
I20250629 01:51:43.907383 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]). Expected version 1 got 0
I20250629 01:51:43.908443 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5dfe921a84694903bbdb3fed3d6f2761
I20250629 01:51:43.909029 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9b82f19de1c74fcd9973b4a4084712e1" } modify_external_catalogs: true
I20250629 01:51:43.909142 1158 ts_tablet_manager.cc:1918] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.909176 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9b82f19de1c74fcd9973b4a4084712e1" } modify_external_catalogs: true
I20250629 01:51:43.909497 1158 log.cc:1199] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a229b4f9d19147a5990f799d1dc0e5d7
I20250629 01:51:43.909895 1158 ts_tablet_manager.cc:1939] T a229b4f9d19147a5990f799d1dc0e5d7 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:43.895306 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 351f4a7ae7e244999dda81d282de505d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 351f4a7ae7e244999dda81d282de505d already in progress: creating tablet
I20250629 01:51:43.910025 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_58a60b60dab64affa9d8b5b1acf53326" } modify_external_catalogs: true
W20250629 01:51:43.910228 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0c712d978a3848b9bb3abdbce8314dec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c712d978a3848b9bb3abdbce8314dec already in progress: creating tablet
I20250629 01:51:43.910310 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 9bbbeb43473a4f6f892056a30f62589f (DEFAULT_TABLE table=default.Table_9245aa65a3734982b3e84187cc176312 [id=8705c82c65df415abf3299b10b90f48c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:43.910404 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e82b1352bb4d49769808dc402c85890f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e82b1352bb4d49769808dc402c85890f already in progress: creating tablet
I20250629 01:51:43.902947 886 tablet_service.cc:1468] Processing CreateTablet for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (DEFAULT_TABLE table=default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:43.910815 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a229b4f9d19147a5990f799d1dc0e5d7 because the tablet was not found. No further retry: Not found: Tablet not found: a229b4f9d19147a5990f799d1dc0e5d7
I20250629 01:51:43.910766 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 (DEFAULT_TABLE table=default.Table_6f6d8cf186fc4934ac5690bac0754373 [id=db3232e7e81240a699e87310dc0627d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.912621 454 catalog_manager.cc:5582] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:43.913447 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]). Expected version 1 got 0
I20250629 01:51:43.914511 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbbeb43473a4f6f892056a30f62589f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.914876 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b (DEFAULT_TABLE table=default.Table_f128352ab4bd4f258d90d9a4022577a8 [id=b516da4148384818bcc69e4259f20ef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.826828 1159 consensus_queue.cc:260] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:43.916260 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (DEFAULT_TABLE table=default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.916947 1158 tablet_replica.cc:331] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:43.917114 1158 raft_consensus.cc:2241] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.917272 1158 raft_consensus.cc:2270] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.917719 1158 ts_tablet_manager.cc:1905] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.918977 1364 raft_consensus.cc:491] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:43.919142 1364 raft_consensus.cc:513] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:43.919783 1364 leader_election.cc:290] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:43.920320 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 (DEFAULT_TABLE table=default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.920758 1017 tablet_service.cc:1468] Processing CreateTablet for tablet ddc24d0e668c49ac825b2578849ab2a3 (DEFAULT_TABLE table=default.Table_adf0dc244a084dcf82f6cb6f22ecc128 [id=d3d515038aa84669a3a64824de88f06e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:43.921411 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4701123dfa467ebb6c95f9742b3874" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:43.929651 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462789e8daef42a08ff8b2cd2ddd1c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.932190 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d21dfd48ea4a95a2fcecde6735974b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.932367 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3a65c9c1c740b1865b96a6e2aea524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.936211 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddc24d0e668c49ac825b2578849ab2a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.936493 449 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_94c8eccb1a094394bad377478edb243d [id=60094a0a7c134476a2a4e44d8555c5d6] alter complete (version 4)
I20250629 01:51:43.939066 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a40c1a8eec9549a8bdf695d6516440a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.945096 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462789e8daef42a08ff8b2cd2ddd1c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:43.945760 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.950570 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_36109461cd234772b6d33074e7fd5a1a" } new_table_name: "default.Table_85587e344a024adea91f5cff5e7317f7" modify_external_catalogs: true
I20250629 01:51:43.950690 1159 ts_tablet_manager.cc:1428] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.104s user 0.002s sys 0.000s
I20250629 01:51:43.957296 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_02ed750488084801a295ba2e0b78d76f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.957499 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a229b4f9d19147a5990f799d1dc0e5d7 (table default.Table_654038c5004a437482989db6feea6573 [id=0c84d077516244288b296db0048f3497]) successfully deleted
W20250629 01:51:43.958389 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02ed750488084801a295ba2e0b78d76f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.958725 1159 tablet_bootstrap.cc:492] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:43.960551 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4701123dfa467ebb6c95f9742b3874" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:43.965345 1159 tablet_bootstrap.cc:654] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:43.967553 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f consensus state has no leader
I20250629 01:51:43.970149 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2f4ee457ae9447697c7e33b84157bd7" } new_table_name: "default.Table_43f2d7b5617b45c9a3acc51f0add89df" modify_external_catalogs: true
I20250629 01:51:43.970496 1157 ts_tablet_manager.cc:1918] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:43.970592 1157 log.cc:1199] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/cade2a7984a3413a9690af48b8c28e84
I20250629 01:51:43.970947 1157 ts_tablet_manager.cc:1939] T cade2a7984a3413a9690af48b8c28e84 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.971351 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9b3067ec3f994a44bf4ecacbc937f414" } new_table_name: "default.Table_d9bb8cc1cab544148faa53db857172d3" modify_external_catalogs: true
I20250629 01:51:43.971649 1157 tablet_replica.cc:331] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:43.971815 1157 raft_consensus.cc:2241] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.971939 1157 raft_consensus.cc:2270] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.973330 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet cade2a7984a3413a9690af48b8c28e84 (table default.Table_cdb1b42c2c774292b4ee2e323ddee5f6 [id=111d4dc551ed44a1a99a4dfcb0916eb5]) successfully deleted
I20250629 01:51:43.974036 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0ed9b0a279de49078207546c6ddbc21f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:43.974213 1157 ts_tablet_manager.cc:1905] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:43.974489 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ed9b0a279de49078207546c6ddbc21f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.980885 1339 tablet.cc:1722] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:43.982868 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9624117e97974a47ac2bfe525da6f405
I20250629 01:51:43.983376 446 master_service.cc:948] ReplaceTablet: received request to replace tablet d3a91740f16744439f55077c4fbadee1 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.983727 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 241a5c9aee3f4297b67b18380ef5ef88
I20250629 01:51:43.984345 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_119622f658584820a8fcd60b2b640e05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.984732 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_119622f658584820a8fcd60b2b640e05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.985666 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 47d4287317ec4788bd96ad174c31748d
I20250629 01:51:43.986186 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e0533c0c40ca4af1be6ac30ee37f3aed" } modify_external_catalogs: true
I20250629 01:51:43.986346 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e0533c0c40ca4af1be6ac30ee37f3aed" } modify_external_catalogs: true
I20250629 01:51:43.960806 775 raft_consensus.cc:2391] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in this term.
I20250629 01:51:43.987604 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca34b2e67ca14f05822bd6615eb5d107" } new_table_name: "default.Table_df8a368b4df84492add9218c8d1deb22" modify_external_catalogs: true
I20250629 01:51:43.961223 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.961310 744 tablet_service.cc:1515] Processing DeleteTablet for tablet caf5826b7e494afc9ae884fe7e94353a with delete_type TABLET_DATA_DELETED (replaced by tablet 81b32bac5a2144d1943285f5f446d53d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.961376 752 tablet_service.cc:1515] Processing DeleteTablet for tablet b939a8568ac14c339258af925fcb809b with delete_type TABLET_DATA_DELETED (replaced by tablet 34172d05ae40417494cd8f67f4cb981e) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.961469 751 tablet_service.cc:1515] Processing DeleteTablet for tablet fb359994b20748518a676045ac6f0df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.961534 746 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.961625 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 02d072bf72044dc48d8090660ecb7d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.961771 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:43.988791 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.966619 1156 ts_tablet_manager.cc:1918] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:43.989470 1156 log.cc:1199] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/3a45e95a4a6a45f9b7c7b09bd69112b4
I20250629 01:51:43.989471 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9624117e97974a47ac2bfe525da6f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:43.989796 1156 ts_tablet_manager.cc:1939] T 3a45e95a4a6a45f9b7c7b09bd69112b4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:43.990317 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_208f2046cc46452887e343bab242e4b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.990708 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_208f2046cc46452887e343bab242e4b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:43.990743 1156 tablet_replica.cc:331] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:43.990908 1156 raft_consensus.cc:2241] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:43.991046 1156 raft_consensus.cc:2270] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:43.980137 1382 tablet.cc:1722] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:43.961838 748 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 with delete_type TABLET_DATA_DELETED (replaced by tablet a2a9664bb7a946c48cf79a09955ecad5) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:43.996249 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f5941dd6381041538be6c92858d6ba13 consensus state has no leader
I20250629 01:51:43.996455 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:43.996510 1156 ts_tablet_manager.cc:1905] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:43.996937 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_32564855276a40cda0182a5675767273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.997316 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32564855276a40cda0182a5675767273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:43.998273 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a90b206d58d04acdb1a8cb0c45d68c31 consensus state has no leader
I20250629 01:51:43.998494 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 0b7a640451934bff8672cd88dc86c529 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:43.999008 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75ce1bd4af434f7e99092aafc0ab0118 consensus state has no leader
I20250629 01:51:43.999300 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_07e9a307bb3344089bc369d3f1bf6c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:43.999711 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07e9a307bb3344089bc369d3f1bf6c34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.000689 1157 ts_tablet_manager.cc:1918] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.000811 1157 log.cc:1199] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d3a91740f16744439f55077c4fbadee1
I20250629 01:51:44.001145 1157 ts_tablet_manager.cc:1939] T d3a91740f16744439f55077c4fbadee1 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:43.960731 757 tablet_service.cc:1468] Processing CreateTablet for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (DEFAULT_TABLE table=default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.002008 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462789e8daef42a08ff8b2cd2ddd1c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.003162 1156 ts_tablet_manager.cc:1918] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.003196 1159 tablet_bootstrap.cc:492] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.003263 1156 log.cc:1199] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/caf5826b7e494afc9ae884fe7e94353a
I20250629 01:51:44.003317 1159 ts_tablet_manager.cc:1397] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.011s
I20250629 01:51:44.003654 1156 ts_tablet_manager.cc:1939] T caf5826b7e494afc9ae884fe7e94353a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.003656 446 master_service.cc:948] ReplaceTablet: received request to replace tablet f5941dd6381041538be6c92858d6ba13 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:44.004081 1159 raft_consensus.cc:357] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.004289 1159 raft_consensus.cc:383] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.004370 1156 tablet_replica.cc:331] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.004360 1159 raft_consensus.cc:738] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.004556 1156 raft_consensus.cc:2241] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.004572 1159 consensus_queue.cc:260] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.004696 1156 raft_consensus.cc:2270] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.005328 1156 ts_tablet_manager.cc:1905] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.005410 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6e4d952c729c4984a21f16da686e28f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.005802 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e4d952c729c4984a21f16da686e28f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.006737 1159 ts_tablet_manager.cc:1428] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:44.007645 1159 tablet_bootstrap.cc:492] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.008690 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d3a91740f16744439f55077c4fbadee1 (table default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 [id=cf0d80c2c9e2464db9ded94438df584a]) successfully deleted
I20250629 01:51:44.008803 1155 tablet_bootstrap.cc:492] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:44.008927 1155 ts_tablet_manager.cc:1397] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.228s user 0.026s sys 0.065s
W20250629 01:51:44.008914 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5dfe921a84694903bbdb3fed3d6f2761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dfe921a84694903bbdb3fed3d6f2761 already in progress: creating tablet
W20250629 01:51:44.009104 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 241a5c9aee3f4297b67b18380ef5ef88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 241a5c9aee3f4297b67b18380ef5ef88 already in progress: creating tablet
I20250629 01:51:44.010068 894 tablet_service.cc:1515] Processing DeleteTablet for tablet fb359994b20748518a676045ac6f0df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.010141 1155 raft_consensus.cc:357] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.010290 1155 raft_consensus.cc:383] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.010290 887 tablet_service.cc:1515] Processing DeleteTablet for tablet caf5826b7e494afc9ae884fe7e94353a with delete_type TABLET_DATA_DELETED (replaced by tablet 81b32bac5a2144d1943285f5f446d53d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.010360 1155 raft_consensus.cc:738] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.010474 884 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.010535 1155 consensus_queue.cc:260] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.010637 894 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 with delete_type TABLET_DATA_DELETED (replaced by tablet a2a9664bb7a946c48cf79a09955ecad5) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.010926 1157 tablet_replica.cc:331] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:44.011008 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.011065 1157 raft_consensus.cc:2241] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.011150 986 leader_election.cc:304] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:44.011204 1157 raft_consensus.cc:2270] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.011534 1364 raft_consensus.cc:3058] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:44.011703 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 47d4287317ec4788bd96ad174c31748d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.011795 1155 ts_tablet_manager.cc:1428] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:44.011790 1157 ts_tablet_manager.cc:1905] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.012017 1155 tablet_bootstrap.cc:492] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:44.013358 1155 tablet_bootstrap.cc:654] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.015790 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 02d072bf72044dc48d8090660ecb7d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:44.015897 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9624117e97974a47ac2bfe525da6f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9624117e97974a47ac2bfe525da6f405 already in progress: creating tablet
I20250629 01:51:44.016000 1159 tablet_bootstrap.cc:654] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.016644 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.016734 1156 ts_tablet_manager.cc:1918] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.016839 1156 log.cc:1199] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b939a8568ac14c339258af925fcb809b
I20250629 01:51:44.017103 1156 ts_tablet_manager.cc:1939] T b939a8568ac14c339258af925fcb809b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.017818 1364 raft_consensus.cc:2747] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:44.018199 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet dd6013bb40c248f29f40eab2c3008f6b with delete_type TABLET_DATA_DELETED (replaced by tablet a485e0e6e2f04aa6bd67a62f6d30be26) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.019425 1157 ts_tablet_manager.cc:1918] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.019490 1158 ts_tablet_manager.cc:1918] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:44.019541 1157 log.cc:1199] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/fb359994b20748518a676045ac6f0df8
I20250629 01:51:44.019618 1158 log.cc:1199] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b6688e9665f24e73b80802d47a80a2cd
I20250629 01:51:44.019915 1157 ts_tablet_manager.cc:1939] T fb359994b20748518a676045ac6f0df8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:44.019995 1158 ts_tablet_manager.cc:1939] T b6688e9665f24e73b80802d47a80a2cd P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.021140 1158 tablet_replica.cc:331] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.021343 1158 raft_consensus.cc:2241] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.021481 1158 raft_consensus.cc:2270] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.022151 1156 tablet_replica.cc:331] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.022353 1156 raft_consensus.cc:2241] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.022521 1156 raft_consensus.cc:2270] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.023191 1387 tablet.cc:1722] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:44.023808 1156 ts_tablet_manager.cc:1905] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.025461 1159 tablet_bootstrap.cc:492] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.025570 1159 ts_tablet_manager.cc:1397] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250629 01:51:44.026136 1159 raft_consensus.cc:357] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.026968 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet fb359994b20748518a676045ac6f0df8 (table default.Table_a23d7cbbf7804d2e95f76b62ec45c158 [id=4428e6c03ff541c89ad394b08004be0b]) successfully deleted
I20250629 01:51:44.027892 1159 raft_consensus.cc:383] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.027967 1159 raft_consensus.cc:738] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.027571 1157 tablet_replica.cc:331] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:44.028255 1157 raft_consensus.cc:2241] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.028954 1427 raft_consensus.cc:491] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.029093 1427 raft_consensus.cc:513] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.029759 1427 leader_election.cc:290] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.029843 1158 ts_tablet_manager.cc:1905] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.030705 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5b835ff8624ebf8956eeb965ce5950" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:44.031270 850 leader_election.cc:343] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.028415 1157 raft_consensus.cc:2270] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.029074 1159 consensus_queue.cc:260] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.032495 1157 ts_tablet_manager.cc:1905] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.032647 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e962a020664b42ea918e5952dc43aade
I20250629 01:51:44.034066 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5b835ff8624ebf8956eeb965ce5950" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:44.034260 775 raft_consensus.cc:2466] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:44.034375 889 tablet_service.cc:1515] Processing DeleteTablet for tablet e962a020664b42ea918e5952dc43aade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.034706 850 leader_election.cc:304] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:44.035266 1427 raft_consensus.cc:2802] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:44.035357 1427 raft_consensus.cc:491] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:44.035431 1427 raft_consensus.cc:3058] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:44.035916 1159 ts_tablet_manager.cc:1428] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:44.036690 1159 tablet_bootstrap.cc:492] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.037883 1427 raft_consensus.cc:513] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.038043 1159 tablet_bootstrap.cc:654] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.038409 1427 leader_election.cc:290] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.039078 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5b835ff8624ebf8956eeb965ce5950" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
W20250629 01:51:44.039559 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cabb0298771e4fd5be29e530d3a8ed3d consensus state has no leader
I20250629 01:51:44.039788 450 master_service.cc:948] ReplaceTablet: received request to replace tablet fca257d6cb614110948e1045dee44bdc from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:44.040063 850 leader_election.cc:343] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.040452 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5b835ff8624ebf8956eeb965ce5950" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:44.040621 775 raft_consensus.cc:3058] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:44.041352 1158 ts_tablet_manager.cc:1918] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.041460 1158 log.cc:1199] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/64477468c6da4b4ba282d8530eb5cbb5
I20250629 01:51:44.041756 1158 ts_tablet_manager.cc:1939] T 64477468c6da4b4ba282d8530eb5cbb5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.042915 775 raft_consensus.cc:2466] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:44.043351 850 leader_election.cc:304] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:44.043745 1427 raft_consensus.cc:2802] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:44.043957 1427 raft_consensus.cc:695] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:44.044559 1427 consensus_queue.cc:237] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.046363 1159 tablet_bootstrap.cc:492] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.046514 1159 ts_tablet_manager.cc:1397] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250629 01:51:44.047200 1155 tablet_bootstrap.cc:492] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:44.047151 1159 raft_consensus.cc:357] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.047322 1159 raft_consensus.cc:383] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.047410 1159 raft_consensus.cc:738] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.047941 1155 ts_tablet_manager.cc:1397] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.003s
I20250629 01:51:44.048367 1154 tablet_bootstrap.cc:492] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.048512 1154 ts_tablet_manager.cc:1397] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.225s user 0.004s sys 0.000s
I20250629 01:51:44.049136 1154 raft_consensus.cc:357] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.049316 1154 raft_consensus.cc:383] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.049291 1159 consensus_queue.cc:260] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.049396 1154 raft_consensus.cc:738] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.049616 1154 consensus_queue.cc:260] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.050022 1154 ts_tablet_manager.cc:1428] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:44.049216 1155 raft_consensus.cc:357] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.050153 1159 ts_tablet_manager.cc:1428] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:44.050274 1155 raft_consensus.cc:383] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.050318 1154 tablet_bootstrap.cc:492] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.050418 1159 tablet_bootstrap.cc:492] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.050514 1155 raft_consensus.cc:738] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.050839 1155 consensus_queue.cc:260] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.051853 1159 tablet_bootstrap.cc:654] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.053820 1430 raft_consensus.cc:491] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.053961 1430 raft_consensus.cc:513] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.054415 1430 leader_election.cc:290] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:44.054466 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet fb359994b20748518a676045ac6f0df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb359994b20748518a676045ac6f0df8 already in progress: creating tablet
I20250629 01:51:44.055181 1157 ts_tablet_manager.cc:1918] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.055284 1157 log.cc:1199] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/caf5826b7e494afc9ae884fe7e94353a
I20250629 01:51:44.055575 1157 ts_tablet_manager.cc:1939] T caf5826b7e494afc9ae884fe7e94353a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:44.056154 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet be8e0946a20340b38835c4d8daeaa7b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8e0946a20340b38835c4d8daeaa7b8 already in progress: creating tablet
W20250629 01:51:44.056377 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0d925ffce89b40258b29ec021bfa5f92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d925ffce89b40258b29ec021bfa5f92 already in progress: creating tablet
I20250629 01:51:44.056418 1157 tablet_replica.cc:331] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:44.056524 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet caf5826b7e494afc9ae884fe7e94353a (table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]) successfully deleted
W20250629 01:51:44.056701 442 catalog_manager.cc:4895] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cade2a7984a3413a9690af48b8c28e84 because the tablet was not found. No further retry: Not found: Tablet not found: cade2a7984a3413a9690af48b8c28e84
I20250629 01:51:44.056797 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f3bf5fd9b674eeb8a5a029152cbae13" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:44.056696 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a485e0e6e2f04aa6bd67a62f6d30be26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a485e0e6e2f04aa6bd67a62f6d30be26 already in progress: creating tablet
W20250629 01:51:44.057319 850 leader_election.cc:343] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.057602 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f3bf5fd9b674eeb8a5a029152cbae13" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:44.058058 850 leader_election.cc:343] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.058161 850 leader_election.cc:304] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:44.058364 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9cf553b291854a6f9e7ed78b5db5f18c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf553b291854a6f9e7ed78b5db5f18c already in progress: creating tablet
I20250629 01:51:44.058398 1430 raft_consensus.cc:2747] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:44.058542 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0a574f95ac2a4193a563e63cbe120768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a574f95ac2a4193a563e63cbe120768 already in progress: creating tablet
W20250629 01:51:44.058734 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e1fe3880e51345269e852014d0d60bb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1fe3880e51345269e852014d0d60bb6 already in progress: creating tablet
W20250629 01:51:44.058916 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 35990d206ba64159ab05ab7b7b19d064 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35990d206ba64159ab05ab7b7b19d064 already in progress: creating tablet
W20250629 01:51:44.059092 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8799b1858bf2408a85f0956578639393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8799b1858bf2408a85f0956578639393 already in progress: creating tablet
I20250629 01:51:44.060019 1158 tablet_replica.cc:331] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.060238 1158 raft_consensus.cc:2241] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.060390 1158 raft_consensus.cc:2270] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.060690 1155 ts_tablet_manager.cc:1428] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250629 01:51:44.063252 1154 tablet_bootstrap.cc:654] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.063983 1155 tablet_bootstrap.cc:492] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:44.065081 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f5941dd6381041538be6c92858d6ba13
I20250629 01:51:44.065493 1159 tablet_bootstrap.cc:492] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.065537 446 catalog_manager.cc:6652] ReplaceTablet: tablet f5941dd6381041538be6c92858d6ba13 deleted and replaced by tablet 347fa6a12a744ee88058643b7143e39d
I20250629 01:51:44.065601 1159 ts_tablet_manager.cc:1397] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20250629 01:51:44.066195 1159 raft_consensus.cc:357] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.066373 1159 raft_consensus.cc:383] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.066452 1159 raft_consensus.cc:738] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.067972 1156 ts_tablet_manager.cc:1918] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.067986 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f5941dd6381041538be6c92858d6ba13 with delete_type TABLET_DATA_DELETED (replaced by tablet 347fa6a12a744ee88058643b7143e39d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.068105 1156 log.cc:1199] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/fb359994b20748518a676045ac6f0df8
I20250629 01:51:44.068212 1159 consensus_queue.cc:260] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.068393 1156 ts_tablet_manager.cc:1939] T fb359994b20748518a676045ac6f0df8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.068419 1158 ts_tablet_manager.cc:1905] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:44.059254 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f8b38c20eb1849b1a9f8acd40315eef5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8b38c20eb1849b1a9f8acd40315eef5 already in progress: creating tablet
I20250629 01:51:44.069017 1159 ts_tablet_manager.cc:1428] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:44.069168 1156 tablet_replica.cc:331] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.069233 1159 tablet_bootstrap.cc:492] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.069325 1156 raft_consensus.cc:2241] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:44.069346 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet fe14d1d54db74a99b05cf8b1a60827a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe14d1d54db74a99b05cf8b1a60827a9 already in progress: creating tablet
I20250629 01:51:44.069439 1156 raft_consensus.cc:2270] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:44.069543 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75fd4ffcf896419b81a1c0eb7c5aeeec already in progress: creating tablet
I20250629 01:51:44.069864 1156 ts_tablet_manager.cc:1905] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.070262 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b6688e9665f24e73b80802d47a80a2cd (table default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]) successfully deleted
W20250629 01:51:44.070457 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet af9ab324ce7c488cbaf6e0f0322b5c58 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af9ab324ce7c488cbaf6e0f0322b5c58 already in progress: creating tablet
I20250629 01:51:44.072857 1159 tablet_bootstrap.cc:654] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.073348 1157 raft_consensus.cc:2241] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:44.073689 1157 pending_rounds.cc:70] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Trying to abort 1 pending ops.
I20250629 01:51:44.073756 1157 pending_rounds.cc:77] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759158969843712 op_type: NO_OP noop_request { }
I20250629 01:51:44.073856 1157 raft_consensus.cc:2887] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:44.073940 1157 raft_consensus.cc:2270] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.075942 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 64477468c6da4b4ba282d8530eb5cbb5 (table default.Table_6f9bb7305f9d42fbb5b12690468aed5f [id=81e46758eee54c7caae081f9232e9071]) successfully deleted
W20250629 01:51:44.076263 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5dfe921a84694903bbdb3fed3d6f2761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dfe921a84694903bbdb3fed3d6f2761 already in progress: creating tablet
I20250629 01:51:44.076952 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 3a45e95a4a6a45f9b7c7b09bd69112b4 (table default.Table_0b441ce248f2448eabf2d7e3cddff976 [id=ba643eccd0e54ce083459bf535db1fe1]) successfully deleted
W20250629 01:51:44.077127 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5dfe921a84694903bbdb3fed3d6f2761 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5dfe921a84694903bbdb3fed3d6f2761 already in progress: creating tablet
I20250629 01:51:44.077392 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet caf5826b7e494afc9ae884fe7e94353a (table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]) successfully deleted
I20250629 01:51:44.077651 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b939a8568ac14c339258af925fcb809b (table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]) successfully deleted
I20250629 01:51:44.077859 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet fb359994b20748518a676045ac6f0df8 (table default.Table_a23d7cbbf7804d2e95f76b62ec45c158 [id=4428e6c03ff541c89ad394b08004be0b]) successfully deleted
I20250629 01:51:44.078730 1155 tablet_bootstrap.cc:654] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:44.078943 1521 consensus_queue.cc:1162] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250629 01:51:44.079574 1157 ts_tablet_manager.cc:1905] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.081485 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet fca257d6cb614110948e1045dee44bdc
I20250629 01:51:44.081826 450 catalog_manager.cc:6652] ReplaceTablet: tablet fca257d6cb614110948e1045dee44bdc deleted and replaced by tablet a6a012ba891949538800581436ef5380
I20250629 01:51:44.085136 1156 ts_tablet_manager.cc:1918] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.085254 1156 log.cc:1199] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/cade2a7984a3413a9690af48b8c28e84
I20250629 01:51:44.085556 1156 ts_tablet_manager.cc:1939] T cade2a7984a3413a9690af48b8c28e84 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.085752 1157 ts_tablet_manager.cc:1918] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:44.085919 1157 log.cc:1199] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/bd5b835ff8624ebf8956eeb965ce5950
I20250629 01:51:44.086263 1157 ts_tablet_manager.cc:1939] T bd5b835ff8624ebf8956eeb965ce5950 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:44.086325 1156 tablet_replica.cc:331] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.086488 1156 raft_consensus.cc:2241] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.087146 1156 raft_consensus.cc:2270] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.087611 1156 ts_tablet_manager.cc:1905] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.088022 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet bd5b835ff8624ebf8956eeb965ce5950 (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
W20250629 01:51:44.088222 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 47d4287317ec4788bd96ad174c31748d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47d4287317ec4788bd96ad174c31748d already in progress: creating tablet
I20250629 01:51:44.089118 1154 tablet_bootstrap.cc:492] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.089242 1154 ts_tablet_manager.cc:1397] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.005s
I20250629 01:51:44.089747 1159 tablet_bootstrap.cc:492] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.090071 1154 raft_consensus.cc:357] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.090221 1154 raft_consensus.cc:383] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.090297 1154 raft_consensus.cc:738] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.090488 1154 consensus_queue.cc:260] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.091037 1154 ts_tablet_manager.cc:1428] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:44.089901 1157 tablet_replica.cc:331] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:44.089968 889 tablet_service.cc:1515] Processing DeleteTablet for tablet fca257d6cb614110948e1045dee44bdc with delete_type TABLET_DATA_DELETED (replaced by tablet a6a012ba891949538800581436ef5380) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.091073 1159 ts_tablet_manager.cc:1397] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250629 01:51:44.091272 1157 raft_consensus.cc:2241] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.092077 1154 tablet_bootstrap.cc:492] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.092142 1157 raft_consensus.cc:2270] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.093092 1157 ts_tablet_manager.cc:1905] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.096658 1159 raft_consensus.cc:357] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.096827 1159 raft_consensus.cc:383] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.096899 1159 raft_consensus.cc:738] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.096905 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:44.097090 1159 consensus_queue.cc:260] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.097340 1154 tablet_bootstrap.cc:654] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.097435 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384] alter complete (version 1)
I20250629 01:51:44.099574 1159 ts_tablet_manager.cc:1428] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250629 01:51:44.099886 1159 tablet_bootstrap.cc:492] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.102547 1159 tablet_bootstrap.cc:654] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.103129 1156 ts_tablet_manager.cc:1918] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.103253 1156 log.cc:1199] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:44.103374 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 with delete_type TABLET_DATA_DELETED (replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.105020 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 9624117e97974a47ac2bfe525da6f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.105557 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 9624117e97974a47ac2bfe525da6f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.106278 1156 ts_tablet_manager.cc:1939] T 02d072bf72044dc48d8090660ecb7d47 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.106405 889 tablet_service.cc:1468] Processing CreateTablet for tablet e5e8b8e4d42f43c78f34102b76b4b11a (DEFAULT_TABLE table=default.Table_d1d0b27c4dae4594a84e9859774c89d6 [id=e991e29b3d9d4dcc96912754565336f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.106791 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e8b8e4d42f43c78f34102b76b4b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.107012 1156 tablet_replica.cc:331] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.107172 1156 raft_consensus.cc:2241] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.107290 1156 raft_consensus.cc:2270] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.108064 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 (DEFAULT_TABLE table=default.Table_f81f54577f464e6bbe68673617d0e0cd [id=d929c72226424c82a6d8c8bac1c3aac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.108426 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db333c60cb3485a9daae316b9f1a8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.106089 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.111548 894 tablet_service.cc:1468] Processing CreateTablet for tablet da93c730451d42bc8df1e582fb3af818 (DEFAULT_TABLE table=default.Table_8bb871e6aa1745a6a3007e6b15ee75cd [id=88472d60f4244733beb9ff0fd6ce9c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.111847 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da93c730451d42bc8df1e582fb3af818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.112169 1156 ts_tablet_manager.cc:1905] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.112274 885 tablet_service.cc:1468] Processing CreateTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 (DEFAULT_TABLE table=default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.112926 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b17172ab8b4e0dad36ebaac776dc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.114038 884 tablet_service.cc:1468] Processing CreateTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (DEFAULT_TABLE table=default.Table_d2344e007e2f4266aa706d32b9465347 [id=5ebf3881fc6a49b7bf063344e3b5cb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.114341 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.115408 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.115712 887 tablet_service.cc:1468] Processing CreateTablet for tablet 2aca26af31ed4bc994f4d62603edc881 (DEFAULT_TABLE table=default.Table_c1cb87a8cbbd4ab5b62e65e5c31e2c95 [id=72ebf1394e9a40faaee64e1b26e6ffe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.115942 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 47d4287317ec4788bd96ad174c31748d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.116008 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:44.116079 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aca26af31ed4bc994f4d62603edc881. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.116459 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 47d4287317ec4788bd96ad174c31748d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.116880 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e962a020664b42ea918e5952dc43aade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.116858 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f81f54577f464e6bbe68673617d0e0cd" } modify_external_catalogs: true
I20250629 01:51:44.117044 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f81f54577f464e6bbe68673617d0e0cd" } modify_external_catalogs: true
I20250629 01:51:44.117379 757 tablet_service.cc:1515] Processing DeleteTablet for tablet e962a020664b42ea918e5952dc43aade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.117853 757 tablet_service.cc:1515] Processing DeleteTablet for tablet f5941dd6381041538be6c92858d6ba13 with delete_type TABLET_DATA_DELETED (replaced by tablet 347fa6a12a744ee88058643b7143e39d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.118295 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet f5941dd6381041538be6c92858d6ba13 with delete_type TABLET_DATA_DELETED (replaced by tablet 347fa6a12a744ee88058643b7143e39d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.120450 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 648fd1ca8f514ea3ba4554452b19933f with delete_type TABLET_DATA_DELETED (replaced by tablet 0b7a640451934bff8672cd88dc86c529) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.121994 895 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.122054 1156 ts_tablet_manager.cc:1918] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.122074 757 tablet_service.cc:1515] Processing DeleteTablet for tablet fca257d6cb614110948e1045dee44bdc with delete_type TABLET_DATA_DELETED (replaced by tablet a6a012ba891949538800581436ef5380) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.122121 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.122159 1156 log.cc:1199] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/200871d1dde144eeb59cc271ce9e4cfe
I20250629 01:51:44.122447 1156 ts_tablet_manager.cc:1939] T 200871d1dde144eeb59cc271ce9e4cfe P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.122401 888 tablet_service.cc:1468] Processing CreateTablet for tablet 1c995ce14c9a4966b842fcba520cf87d (DEFAULT_TABLE table=default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.122534 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet fca257d6cb614110948e1045dee44bdc with delete_type TABLET_DATA_DELETED (replaced by tablet a6a012ba891949538800581436ef5380) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.122905 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c995ce14c9a4966b842fcba520cf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.123142 1156 tablet_replica.cc:331] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.123860 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e5e8b8e4d42f43c78f34102b76b4b11a (DEFAULT_TABLE table=default.Table_d1d0b27c4dae4594a84e9859774c89d6 [id=e991e29b3d9d4dcc96912754565336f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.124085 891 tablet_service.cc:1468] Processing CreateTablet for tablet 52be184d0480495a894442d6f72a0f12 (DEFAULT_TABLE table=default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.124449 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52be184d0480495a894442d6f72a0f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.124552 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e8b8e4d42f43c78f34102b76b4b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.125698 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0b7a640451934bff8672cd88dc86c529
I20250629 01:51:44.126088 451 catalog_manager.cc:6652] ReplaceTablet: tablet 0b7a640451934bff8672cd88dc86c529 deleted and replaced by tablet d456410cd09845dabd979c123f70741b
I20250629 01:51:44.126713 1156 raft_consensus.cc:2241] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.126868 1156 raft_consensus.cc:2270] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.127250 1156 ts_tablet_manager.cc:1905] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.129212 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7a640451934bff8672cd88dc86c529 with delete_type TABLET_DATA_DELETED (replaced by tablet d456410cd09845dabd979c123f70741b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.132262 1156 ts_tablet_manager.cc:1918] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.132350 1156 log.cc:1199] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/bd5b835ff8624ebf8956eeb965ce5950
I20250629 01:51:44.132639 1156 ts_tablet_manager.cc:1939] T bd5b835ff8624ebf8956eeb965ce5950 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.132616 895 tablet_service.cc:1468] Processing CreateTablet for tablet e1e22a0c02054c7db514eefe20ee3311 (DEFAULT_TABLE table=default.Table_ed13689efff54739bc1cd029699b4384 [id=26e9b4aa4f6e4c07a700e299dd461e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.132905 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e22a0c02054c7db514eefe20ee3311. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.133407 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:44.134302 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 (DEFAULT_TABLE table=default.Table_f81f54577f464e6bbe68673617d0e0cd [id=d929c72226424c82a6d8c8bac1c3aac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.134621 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db333c60cb3485a9daae316b9f1a8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.136633 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.137341 882 tablet_service.cc:1468] Processing CreateTablet for tablet 47cbc958eca14ed69fb45c3c2984aeef (DEFAULT_TABLE table=default.Table_4080248559414e1b92597c0367d2530c [id=0ae8d45bb2a84f9588953b68b329f929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.138545 893 tablet_service.cc:1468] Processing CreateTablet for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (DEFAULT_TABLE table=default.Table_7fec8ba2ba174536baceb16beaa50ab8 [id=658b17cdd5f649119d2c3dd59b8c619a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.138736 883 tablet_service.cc:1468] Processing CreateTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec (DEFAULT_TABLE table=default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.139544 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 (DEFAULT_TABLE table=default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.139854 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b17172ab8b4e0dad36ebaac776dc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.141022 748 tablet_service.cc:1468] Processing CreateTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 (DEFAULT_TABLE table=default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.141350 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b17172ab8b4e0dad36ebaac776dc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.142414 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.143724 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cbc958eca14ed69fb45c3c2984aeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.143946 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0b9afb9976541b2bc7ab188f35a62ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.145632 890 tablet_service.cc:1468] Processing CreateTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 (DEFAULT_TABLE table=default.Table_afb98d5207ae4d2d8b97b8577ae3a157 [id=e874833b32de4076921fdf0e2f103a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.147536 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648b0a0f742b4e7da9eaff6bd8795fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.148674 757 tablet_service.cc:1468] Processing CreateTablet for tablet e5e8b8e4d42f43c78f34102b76b4b11a (DEFAULT_TABLE table=default.Table_d1d0b27c4dae4594a84e9859774c89d6 [id=e991e29b3d9d4dcc96912754565336f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.149019 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e8b8e4d42f43c78f34102b76b4b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.154176 1158 ts_tablet_manager.cc:1918] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.154381 1158 log.cc:1199] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/dd6013bb40c248f29f40eab2c3008f6b
I20250629 01:51:44.154780 1158 ts_tablet_manager.cc:1939] T dd6013bb40c248f29f40eab2c3008f6b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.156739 748 tablet_service.cc:1468] Processing CreateTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 (DEFAULT_TABLE table=default.Table_f81f54577f464e6bbe68673617d0e0cd [id=d929c72226424c82a6d8c8bac1c3aac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.157159 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db333c60cb3485a9daae316b9f1a8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.161506 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0ba9eabd06174b87a1485f4921895c5f (DEFAULT_TABLE table=default.Table_6e4d952c729c4984a21f16da686e28f0 [id=2a01904ec4bd4b48b528d7c0cfca4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.162220 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba9eabd06174b87a1485f4921895c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.162595 1017 tablet_service.cc:1468] Processing CreateTablet for tablet da93c730451d42bc8df1e582fb3af818 (DEFAULT_TABLE table=default.Table_8bb871e6aa1745a6a3007e6b15ee75cd [id=88472d60f4244733beb9ff0fd6ce9c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.162882 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da93c730451d42bc8df1e582fb3af818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.169009 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7ac85c469b3c4de2aa2320e2261b0dcd" } new_table_name: "default.Table_eca0690b56fd472e8b2baf252d3721ca" modify_external_catalogs: true
I20250629 01:51:44.169329 892 tablet_service.cc:1468] Processing CreateTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 (DEFAULT_TABLE table=default.Table_119622f658584820a8fcd60b2b640e05 [id=c2c6ad07071540d09fb8e8e13d4e5bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.169486 1155 tablet_bootstrap.cc:492] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:44.169610 1155 ts_tablet_manager.cc:1397] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.106s user 0.021s sys 0.037s
I20250629 01:51:44.170078 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f38ccc485043ba8b8a6f644a3e4f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.170279 1155 raft_consensus.cc:357] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.170416 1155 raft_consensus.cc:383] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.170490 1155 raft_consensus.cc:738] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.170703 1155 consensus_queue.cc:260] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.171200 1155 ts_tablet_manager.cc:1428] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:44.171942 753 tablet_service.cc:1468] Processing CreateTablet for tablet da93c730451d42bc8df1e582fb3af818 (DEFAULT_TABLE table=default.Table_8bb871e6aa1745a6a3007e6b15ee75cd [id=88472d60f4244733beb9ff0fd6ce9c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.173060 742 tablet_service.cc:1468] Processing CreateTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (DEFAULT_TABLE table=default.Table_d2344e007e2f4266aa706d32b9465347 [id=5ebf3881fc6a49b7bf063344e3b5cb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.173637 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 2aca26af31ed4bc994f4d62603edc881 (DEFAULT_TABLE table=default.Table_c1cb87a8cbbd4ab5b62e65e5c31e2c95 [id=72ebf1394e9a40faaee64e1b26e6ffe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.173938 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aca26af31ed4bc994f4d62603edc881. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.173946 896 tablet_service.cc:1468] Processing CreateTablet for tablet 38f831a6dd68493382d5f2920d0794a8 (DEFAULT_TABLE table=default.Table_32564855276a40cda0182a5675767273 [id=60589a5a691248a48e89eed5ba92c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.174430 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ea8b3d1cbc0f46d6acfe2b800a5290a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.174805 879 tablet_service.cc:1468] Processing CreateTablet for tablet 95247cc778504f4d85d9013f84c59e69 (DEFAULT_TABLE table=default.Table_815cc062f73f433da58e8a2fcc368854 [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.174893 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea8b3d1cbc0f46d6acfe2b800a5290a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.174999 1017 tablet_service.cc:1468] Processing CreateTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (DEFAULT_TABLE table=default.Table_d2344e007e2f4266aa706d32b9465347 [id=5ebf3881fc6a49b7bf063344e3b5cb40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.175204 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95247cc778504f4d85d9013f84c59e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.175311 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.175890 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_380dba3955bd458caa9a65e48c00f407"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.176720 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_380dba3955bd458caa9a65e48c00f407 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.177528 1159 tablet_bootstrap.cc:492] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.177652 1159 ts_tablet_manager.cc:1397] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.002s
I20250629 01:51:44.178341 1159 raft_consensus.cc:357] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.178514 1159 raft_consensus.cc:383] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.178591 1159 raft_consensus.cc:738] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.178788 1159 consensus_queue.cc:260] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.179211 1159 ts_tablet_manager.cc:1428] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:44.179423 746 tablet_service.cc:1468] Processing CreateTablet for tablet 2aca26af31ed4bc994f4d62603edc881 (DEFAULT_TABLE table=default.Table_c1cb87a8cbbd4ab5b62e65e5c31e2c95 [id=72ebf1394e9a40faaee64e1b26e6ffe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.179495 1159 tablet_bootstrap.cc:492] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.180047 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 582e063f97ab415cab126845789e6e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.180341 1155 tablet_bootstrap.cc:492] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:44.180636 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet cade2a7984a3413a9690af48b8c28e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.181195 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da93c730451d42bc8df1e582fb3af818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.181777 1155 tablet_bootstrap.cc:654] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:44.183990 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d456410cd09845dabd979c123f70741b consensus state has no leader
I20250629 01:51:44.184926 1156 tablet_replica.cc:331] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.185087 1156 raft_consensus.cc:2241] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.185215 1156 raft_consensus.cc:2270] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.185746 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aca26af31ed4bc994f4d62603edc881. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.185813 1156 ts_tablet_manager.cc:1905] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.187335 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.187898 1154 tablet_bootstrap.cc:492] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.188145 1154 ts_tablet_manager.cc:1397] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.097s user 0.002s sys 0.001s
I20250629 01:51:44.188745 1154 raft_consensus.cc:357] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.188917 1154 raft_consensus.cc:383] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.188975 1154 raft_consensus.cc:738] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.189188 1154 consensus_queue.cc:260] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.189594 1154 ts_tablet_manager.cc:1428] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:44.189867 1154 tablet_bootstrap.cc:492] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.191962 1154 tablet_bootstrap.cc:654] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.195722 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e69e519c24094e298cdcd58c4567f525" } modify_external_catalogs: true
I20250629 01:51:44.195873 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e69e519c24094e298cdcd58c4567f525" } modify_external_catalogs: true
I20250629 01:51:44.196548 1156 ts_tablet_manager.cc:1918] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.196653 1156 log.cc:1199] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/582e063f97ab415cab126845789e6e20
I20250629 01:51:44.196939 1156 ts_tablet_manager.cc:1939] T 582e063f97ab415cab126845789e6e20 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.196926 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f6f27d5499af49359886879547b0ac31" } modify_external_catalogs: true
I20250629 01:51:44.197077 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f6f27d5499af49359886879547b0ac31" } modify_external_catalogs: true
I20250629 01:51:44.198016 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5b835ff8624ebf8956eeb965ce5950 with delete_type TABLET_DATA_DELETED (replaced by tablet a2a9664bb7a946c48cf79a09955ecad5) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.200327 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_328919206d1e44838bc19f8a2ff9d371" } modify_external_catalogs: true
I20250629 01:51:44.200475 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_328919206d1e44838bc19f8a2ff9d371" } modify_external_catalogs: true
I20250629 01:51:44.201905 1158 tablet_replica.cc:331] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.202147 1158 raft_consensus.cc:2241] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.202303 1158 raft_consensus.cc:2270] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.204038 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f831a6dd68493382d5f2920d0794a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.204308 879 tablet_service.cc:1468] Processing CreateTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 (DEFAULT_TABLE table=default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.204648 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b60431b72e4792aa1c32fdee40cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.204787 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_81bc81a3198343948ffbd72f6d683481" } new_table_name: "default.Table_10a0ae1fcff049dda2c11881c3191fa3" modify_external_catalogs: true
I20250629 01:51:44.205648 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1c995ce14c9a4966b842fcba520cf87d (DEFAULT_TABLE table=default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.205789 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_94c8eccb1a094394bad377478edb243d" } new_table_name: "default.Table_ab9e746444a24266aca5439c51b30a27" modify_external_catalogs: true
I20250629 01:51:44.205931 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c995ce14c9a4966b842fcba520cf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.206769 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9245aa65a3734982b3e84187cc176312" } new_table_name: "default.Table_a00a094ccfcf45879959855165452687" modify_external_catalogs: true
I20250629 01:51:44.207175 886 tablet_service.cc:1468] Processing CreateTablet for tablet a6a012ba891949538800581436ef5380 (DEFAULT_TABLE table=default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.207471 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a012ba891949538800581436ef5380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.210925 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet cade2a7984a3413a9690af48b8c28e84 (table default.Table_cdb1b42c2c774292b4ee2e323ddee5f6 [id=111d4dc551ed44a1a99a4dfcb0916eb5]) successfully deleted
I20250629 01:51:44.211131 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 02d072bf72044dc48d8090660ecb7d47 (table default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]) successfully deleted
I20250629 01:51:44.211298 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 200871d1dde144eeb59cc271ce9e4cfe (table default.Table_3ac2dc2474da47f7ba25608e622e9a68 [id=5b172406f58f4023b2b76511690541f6]) successfully deleted
I20250629 01:51:44.211467 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet bd5b835ff8624ebf8956eeb965ce5950 (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
W20250629 01:51:44.211654 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9624117e97974a47ac2bfe525da6f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9624117e97974a47ac2bfe525da6f405 already in progress: creating tablet
W20250629 01:51:44.211850 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 241a5c9aee3f4297b67b18380ef5ef88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 241a5c9aee3f4297b67b18380ef5ef88 already in progress: creating tablet
W20250629 01:51:44.212016 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 47d4287317ec4788bd96ad174c31748d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47d4287317ec4788bd96ad174c31748d already in progress: creating tablet
W20250629 01:51:44.212235 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e962a020664b42ea918e5952dc43aade because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e962a020664b42ea918e5952dc43aade already in progress: creating tablet
W20250629 01:51:44.212414 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f5941dd6381041538be6c92858d6ba13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5941dd6381041538be6c92858d6ba13 already in progress: creating tablet
W20250629 01:51:44.212589 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet fca257d6cb614110948e1045dee44bdc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fca257d6cb614110948e1045dee44bdc already in progress: creating tablet
I20250629 01:51:44.212792 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a6a012ba891949538800581436ef5380
I20250629 01:51:44.213050 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 582e063f97ab415cab126845789e6e20 (table default.Table_30a1da646c914cde85078e3e1270a01e [id=26e8b18cac084e4c9e603950a6595c35]) successfully deleted
I20250629 01:51:44.212412 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:44.213917 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1c995ce14c9a4966b842fcba520cf87d (DEFAULT_TABLE table=default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.214279 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c995ce14c9a4966b842fcba520cf87d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.216162 1364 raft_consensus.cc:491] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.216305 1364 raft_consensus.cc:513] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.216890 1159 tablet_bootstrap.cc:654] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.216871 1364 leader_election.cc:290] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:44.217453 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:44.217648 775 raft_consensus.cc:2391] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in this term.
I20250629 01:51:44.218393 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.218791 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.221490 1158 ts_tablet_manager.cc:1905] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:44.222826 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bbbeb43473a4f6f892056a30f62589f consensus state has no leader
I20250629 01:51:44.223304 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 88031f8d8cbc44e4b8c53d9cd5360552
I20250629 01:51:44.223619 746 tablet_service.cc:1468] Processing CreateTablet for tablet 52be184d0480495a894442d6f72a0f12 (DEFAULT_TABLE table=default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.223704 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7a3f650c91894a0090838ca98207ee9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.224134 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52be184d0480495a894442d6f72a0f12. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.224226 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a3f650c91894a0090838ca98207ee9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.224563 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_338563bee1704636a7021bf819690974"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.225028 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_338563bee1704636a7021bf819690974 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.226509 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7a640451934bff8672cd88dc86c529 with delete_type TABLET_DATA_DELETED (replaced by tablet d456410cd09845dabd979c123f70741b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.227051 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7a640451934bff8672cd88dc86c529 with delete_type TABLET_DATA_DELETED (replaced by tablet d456410cd09845dabd979c123f70741b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.227093 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_401301df62574f26a5ca95acde130d41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.227380 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 52be184d0480495a894442d6f72a0f12 (DEFAULT_TABLE table=default.Table_661f7318ce384b5bad0d019f704d12c2 [id=fadb6eabbead48b1a11795ffed0c4d81]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.227494 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_401301df62574f26a5ca95acde130d41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.227684 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52be184d0480495a894442d6f72a0f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.227854 1013 tablet_service.cc:1468] Processing CreateTablet for tablet e1e22a0c02054c7db514eefe20ee3311 (DEFAULT_TABLE table=default.Table_ed13689efff54739bc1cd029699b4384 [id=26e9b4aa4f6e4c07a700e299dd461e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.228416 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e22a0c02054c7db514eefe20ee3311. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.230414 748 tablet_service.cc:1468] Processing CreateTablet for tablet e1e22a0c02054c7db514eefe20ee3311 (DEFAULT_TABLE table=default.Table_ed13689efff54739bc1cd029699b4384 [id=26e9b4aa4f6e4c07a700e299dd461e1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.230789 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1e22a0c02054c7db514eefe20ee3311. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.235162 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1bc3d470df0d4814a655bcff68c956e2 (DEFAULT_TABLE table=default.Table_ecc7812932cb496d8e92eee3befdce44 [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.235597 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b16b6840c24484b90218fffc1da9621 consensus state has no leader
I20250629 01:51:44.235780 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.236233 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d9bb8cc1cab544148faa53db857172d3" } modify_external_catalogs: true
I20250629 01:51:44.236399 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.236907 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d9bb8cc1cab544148faa53db857172d3" } modify_external_catalogs: true
I20250629 01:51:44.237211 1159 tablet_bootstrap.cc:492] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.237301 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 47cbc958eca14ed69fb45c3c2984aeef (DEFAULT_TABLE table=default.Table_4080248559414e1b92597c0367d2530c [id=0ae8d45bb2a84f9588953b68b329f929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.237643 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cbc958eca14ed69fb45c3c2984aeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.242271 893 tablet_service.cc:1468] Processing CreateTablet for tablet 347fa6a12a744ee88058643b7143e39d (DEFAULT_TABLE table=default.Table_85587e344a024adea91f5cff5e7317f7 [id=0fb1390e9626454eba8e8d5c634e3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.245471 1158 ts_tablet_manager.cc:1918] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.245656 1158 log.cc:1199] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/cade2a7984a3413a9690af48b8c28e84
I20250629 01:51:44.246423 1154 tablet_bootstrap.cc:492] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.246645 1154 ts_tablet_manager.cc:1397] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.002s
I20250629 01:51:44.247241 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_85587e344a024adea91f5cff5e7317f7" } new_table_name: "default.Table_37e9efc823274429a519ff980c6caebb" modify_external_catalogs: true
I20250629 01:51:44.247963 1154 raft_consensus.cc:357] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.248169 1154 raft_consensus.cc:383] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.248291 1154 raft_consensus.cc:738] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.248551 1154 consensus_queue.cc:260] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.249711 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_74ed2fc8d5de4c46bbce4736df36a2be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.250123 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ed2fc8d5de4c46bbce4736df36a2be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.253937 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet dd6013bb40c248f29f40eab2c3008f6b (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:51:44.254170 753 tablet_service.cc:1468] Processing CreateTablet for tablet 47cbc958eca14ed69fb45c3c2984aeef (DEFAULT_TABLE table=default.Table_4080248559414e1b92597c0367d2530c [id=0ae8d45bb2a84f9588953b68b329f929]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.254295 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9624117e97974a47ac2bfe525da6f405 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9624117e97974a47ac2bfe525da6f405 already in progress: creating tablet
I20250629 01:51:44.254495 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47cbc958eca14ed69fb45c3c2984aeef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.254495 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 241a5c9aee3f4297b67b18380ef5ef88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 241a5c9aee3f4297b67b18380ef5ef88 already in progress: creating tablet
W20250629 01:51:44.254663 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 47d4287317ec4788bd96ad174c31748d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47d4287317ec4788bd96ad174c31748d already in progress: creating tablet
W20250629 01:51:44.254861 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e962a020664b42ea918e5952dc43aade because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e962a020664b42ea918e5952dc43aade already in progress: creating tablet
W20250629 01:51:44.255041 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f5941dd6381041538be6c92858d6ba13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5941dd6381041538be6c92858d6ba13 already in progress: creating tablet
W20250629 01:51:44.255924 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0b7a640451934bff8672cd88dc86c529 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7a640451934bff8672cd88dc86c529 already in progress: creating tablet
I20250629 01:51:44.257277 742 tablet_service.cc:1468] Processing CreateTablet for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (DEFAULT_TABLE table=default.Table_7fec8ba2ba174536baceb16beaa50ab8 [id=658b17cdd5f649119d2c3dd59b8c619a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.257582 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.260854 740 tablet_service.cc:1468] Processing CreateTablet for tablet 1bc3d470df0d4814a655bcff68c956e2 (DEFAULT_TABLE table=default.Table_ecc7812932cb496d8e92eee3befdce44 [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.261214 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bc3d470df0d4814a655bcff68c956e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.264565 738 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.266330 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9b58ddddb14343dcb197eff4c517689c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.266783 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b58ddddb14343dcb197eff4c517689c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.268508 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a6a012ba891949538800581436ef5380 (DEFAULT_TABLE table=default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.268812 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a012ba891949538800581436ef5380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.268975 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec (DEFAULT_TABLE table=default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.269243 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0b9afb9976541b2bc7ab188f35a62ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.269887 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 (DEFAULT_TABLE table=default.Table_afb98d5207ae4d2d8b97b8577ae3a157 [id=e874833b32de4076921fdf0e2f103a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.270159 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648b0a0f742b4e7da9eaff6bd8795fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.270243 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (DEFAULT_TABLE table=default.Table_7fec8ba2ba174536baceb16beaa50ab8 [id=658b17cdd5f649119d2c3dd59b8c619a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.270722 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.270771 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 0ba9eabd06174b87a1485f4921895c5f (DEFAULT_TABLE table=default.Table_6e4d952c729c4984a21f16da686e28f0 [id=2a01904ec4bd4b48b528d7c0cfca4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.271057 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba9eabd06174b87a1485f4921895c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.271905 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 (DEFAULT_TABLE table=default.Table_208f2046cc46452887e343bab242e4b2 [id=77c157c54c864ddbb4678e9af6b1fe43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.272207 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2845d8fbf3f34c769027b9b6bb3e1f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.272812 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 (DEFAULT_TABLE table=default.Table_119622f658584820a8fcd60b2b640e05 [id=c2c6ad07071540d09fb8e8e13d4e5bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.273090 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f38ccc485043ba8b8a6f644a3e4f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.273784 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 95247cc778504f4d85d9013f84c59e69 (DEFAULT_TABLE table=default.Table_815cc062f73f433da58e8a2fcc368854 [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.274068 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95247cc778504f4d85d9013f84c59e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.274732 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 (DEFAULT_TABLE table=default.Table_0ed9b0a279de49078207546c6ddbc21f [id=4457fae28052479e81efac2f717c1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.275048 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a880c9bb4b94d3b9b12f28d48565f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.275867 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d456410cd09845dabd979c123f70741b (DEFAULT_TABLE table=default.Table_7ac85c469b3c4de2aa2320e2261b0dcd [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.276181 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 347fa6a12a744ee88058643b7143e39d consensus state has no leader
I20250629 01:51:44.276196 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d456410cd09845dabd979c123f70741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.276825 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 9c799a04c0f74492b9fb674f9b0da240 (DEFAULT_TABLE table=default.Table_4799f303eeb041edb1117ae3caa45b62 [id=297fd4ccf3cb46c283189b67c1b8e648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.277084 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c799a04c0f74492b9fb674f9b0da240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.277679 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 (DEFAULT_TABLE table=default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.277949 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 1bc3d470df0d4814a655bcff68c956e2 (DEFAULT_TABLE table=default.Table_ecc7812932cb496d8e92eee3befdce44 [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.278251 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bc3d470df0d4814a655bcff68c956e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.278934 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 347fa6a12a744ee88058643b7143e39d (DEFAULT_TABLE table=default.Table_85587e344a024adea91f5cff5e7317f7 [id=0fb1390e9626454eba8e8d5c634e3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.279224 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347fa6a12a744ee88058643b7143e39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.279837 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 02d072bf72044dc48d8090660ecb7d47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.280001 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.280167 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.280326 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 88031f8d8cbc44e4b8c53d9cd5360552 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.281426 755 tablet_service.cc:1468] Processing CreateTablet for tablet d456410cd09845dabd979c123f70741b (DEFAULT_TABLE table=default.Table_7ac85c469b3c4de2aa2320e2261b0dcd [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.281718 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d456410cd09845dabd979c123f70741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.284754 1154 ts_tablet_manager.cc:1428] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.038s user 0.000s sys 0.001s
I20250629 01:51:44.284899 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 75ce1bd4af434f7e99092aafc0ab0118
I20250629 01:51:44.291553 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_03045b3c97554436b5b29949fe6a2ccb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.291921 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03045b3c97554436b5b29949fe6a2ccb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.296314 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d34c7a4da774448f8a7e20900b74c956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.300825 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d34c7a4da774448f8a7e20900b74c956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.301384 775 raft_consensus.cc:1273] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:44.301959 1364 consensus_queue.cc:1035] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:44.303630 1158 ts_tablet_manager.cc:1939] T cade2a7984a3413a9690af48b8c28e84 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.308327 1158 tablet_replica.cc:331] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.310334 1158 raft_consensus.cc:2241] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.310910 1158 raft_consensus.cc:2270] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.313367 1158 ts_tablet_manager.cc:1905] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.271654 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 38f831a6dd68493382d5f2920d0794a8 (DEFAULT_TABLE table=default.Table_32564855276a40cda0182a5675767273 [id=60589a5a691248a48e89eed5ba92c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.320533 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f831a6dd68493382d5f2920d0794a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.330133 1158 ts_tablet_manager.cc:1918] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.330384 1158 log.cc:1199] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/bd5b835ff8624ebf8956eeb965ce5950
I20250629 01:51:44.330685 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8f9f59fbe9634ac8b27f990d9496f637 (DEFAULT_TABLE table=default.Table_02ed750488084801a295ba2e0b78d76f [id=167bdf6c439f47ddb772b1b6c244473c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.331007 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f9f59fbe9634ac8b27f990d9496f637. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.336925 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5f4d9b39ee1546eca5a1fb0701b2b93f" } modify_external_catalogs: true
I20250629 01:51:44.337116 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5f4d9b39ee1546eca5a1fb0701b2b93f" } modify_external_catalogs: true
I20250629 01:51:44.183030 1157 ts_tablet_manager.cc:1918] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.339665 880 tablet_service.cc:1468] Processing CreateTablet for tablet 8f9f59fbe9634ac8b27f990d9496f637 (DEFAULT_TABLE table=default.Table_02ed750488084801a295ba2e0b78d76f [id=167bdf6c439f47ddb772b1b6c244473c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.340044 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 (DEFAULT_TABLE table=default.Table_0ed9b0a279de49078207546c6ddbc21f [id=4457fae28052479e81efac2f717c1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.342795 1154 tablet_bootstrap.cc:492] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.343426 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b60431b72e4792aa1c32fdee40cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.343499 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:44.344110 1427 raft_consensus.cc:491] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.346614 752 tablet_service.cc:1468] Processing CreateTablet for tablet 0ba9eabd06174b87a1485f4921895c5f (DEFAULT_TABLE table=default.Table_6e4d952c729c4984a21f16da686e28f0 [id=2a01904ec4bd4b48b528d7c0cfca4a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.346982 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ba9eabd06174b87a1485f4921895c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.347898 741 tablet_service.cc:1468] Processing CreateTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 (DEFAULT_TABLE table=default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.348634 1430 raft_consensus.cc:491] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.349293 889 tablet_service.cc:1468] Processing CreateTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 (DEFAULT_TABLE table=default.Table_208f2046cc46452887e343bab242e4b2 [id=77c157c54c864ddbb4678e9af6b1fe43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.351292 744 tablet_service.cc:1468] Processing CreateTablet for tablet 38f831a6dd68493382d5f2920d0794a8 (DEFAULT_TABLE table=default.Table_32564855276a40cda0182a5675767273 [id=60589a5a691248a48e89eed5ba92c6d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.351627 754 tablet_service.cc:1468] Processing CreateTablet for tablet 9c799a04c0f74492b9fb674f9b0da240 (DEFAULT_TABLE table=default.Table_4799f303eeb041edb1117ae3caa45b62 [id=297fd4ccf3cb46c283189b67c1b8e648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.264401 739 tablet_service.cc:1468] Processing CreateTablet for tablet 347fa6a12a744ee88058643b7143e39d (DEFAULT_TABLE table=default.Table_85587e344a024adea91f5cff5e7317f7 [id=0fb1390e9626454eba8e8d5c634e3943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.352624 745 tablet_service.cc:1468] Processing CreateTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 (DEFAULT_TABLE table=default.Table_0ed9b0a279de49078207546c6ddbc21f [id=4457fae28052479e81efac2f717c1bb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.353824 749 tablet_service.cc:1468] Processing CreateTablet for tablet a6a012ba891949538800581436ef5380 (DEFAULT_TABLE table=default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.351517 751 tablet_service.cc:1468] Processing CreateTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 (DEFAULT_TABLE table=default.Table_afb98d5207ae4d2d8b97b8577ae3a157 [id=e874833b32de4076921fdf0e2f103a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.355150 750 tablet_service.cc:1468] Processing CreateTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 (DEFAULT_TABLE table=default.Table_119622f658584820a8fcd60b2b640e05 [id=c2c6ad07071540d09fb8e8e13d4e5bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.358479 756 tablet_service.cc:1468] Processing CreateTablet for tablet 95247cc778504f4d85d9013f84c59e69 (DEFAULT_TABLE table=default.Table_815cc062f73f433da58e8a2fcc368854 [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.355546 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 (DEFAULT_TABLE table=default.Table_208f2046cc46452887e343bab242e4b2 [id=77c157c54c864ddbb4678e9af6b1fe43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.360795 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8f9f59fbe9634ac8b27f990d9496f637 (DEFAULT_TABLE table=default.Table_02ed750488084801a295ba2e0b78d76f [id=167bdf6c439f47ddb772b1b6c244473c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.361383 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b60431b72e4792aa1c32fdee40cdb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.362982 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.369817 1157 log.cc:1199] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:44.369949 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.370302 1387 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:44.372815 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2ad265c77d0047c38430242c53182e01 (DEFAULT_TABLE table=default.Table_ea8b3d1cbc0f46d6acfe2b800a5290a3 [id=7425128540f048519c623976fd1aaaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.373210 883 tablet_service.cc:1468] Processing CreateTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 (DEFAULT_TABLE table=default.Table_380dba3955bd458caa9a65e48c00f407 [id=77b6e244496149c5a3e46fdfb5bbe791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.378155 895 tablet_service.cc:1468] Processing CreateTablet for tablet 9c799a04c0f74492b9fb674f9b0da240 (DEFAULT_TABLE table=default.Table_4799f303eeb041edb1117ae3caa45b62 [id=297fd4ccf3cb46c283189b67c1b8e648]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.384938 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bc3d470df0d4814a655bcff68c956e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.385339 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_921f95200675447782407805dfd9fd4a" } new_table_name: "default.Table_b39b700fff9245d8a067319ab01e5d8e" modify_external_catalogs: true
I20250629 01:51:44.385707 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 88031f8d8cbc44e4b8c53d9cd5360552 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.389312 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d6363385f77141019261e5492e744cbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.389797 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6363385f77141019261e5492e744cbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.390165 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f2175d9af5064903ab1c97c2035e2527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.390694 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2175d9af5064903ab1c97c2035e2527 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.393852 1430 raft_consensus.cc:513] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.395066 1382 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:44.396540 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2845d8fbf3f34c769027b9b6bb3e1f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.396884 1159 ts_tablet_manager.cc:1397] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.003s
I20250629 01:51:44.398706 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.399070 885 tablet_service.cc:1468] Processing CreateTablet for tablet 82a837932f14419c8b32fcd2c9358215 (DEFAULT_TABLE table=default.Table_401301df62574f26a5ca95acde130d41 [id=ed3c8fa707924f8ea53d590a7f9b3d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.399247 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 88031f8d8cbc44e4b8c53d9cd5360552 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.395967 1427 raft_consensus.cc:513] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.399451 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f9f59fbe9634ac8b27f990d9496f637. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.399890 1427 leader_election.cc:290] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:44.401271 890 tablet_service.cc:1468] Processing CreateTablet for tablet 9cfff25bea764e62954e49611b71703c (DEFAULT_TABLE table=default.Table_7a3f650c91894a0090838ca98207ee9d [id=875222c495154458ae2052f4ca5d220f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.404547 1159 raft_consensus.cc:357] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.404765 757 tablet_service.cc:1468] Processing CreateTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec (DEFAULT_TABLE table=default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.405462 1156 tablet_replica.cc:331] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.405655 1156 raft_consensus.cc:2241] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.405849 1156 raft_consensus.cc:2270] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.406351 1156 ts_tablet_manager.cc:1905] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.407845 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2845d8fbf3f34c769027b9b6bb3e1f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.407994 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f9f59fbe9634ac8b27f990d9496f637. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.408751 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a880c9bb4b94d3b9b12f28d48565f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.409778 1430 leader_election.cc:290] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:44.410889 1339 tablet.cc:1722] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:44.397010 877 tablet_service.cc:1468] Processing CreateTablet for tablet d456410cd09845dabd979c123f70741b (DEFAULT_TABLE table=default.Table_7ac85c469b3c4de2aa2320e2261b0dcd [id=e735a7d5da48458da86326bf01640f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.411970 1157 ts_tablet_manager.cc:1939] T 02d072bf72044dc48d8090660ecb7d47 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:44.413364 887 tablet_service.cc:1468] Processing CreateTablet for tablet 2249d5cba1544653b088312e7d74802e (DEFAULT_TABLE table=default.Table_338563bee1704636a7021bf819690974 [id=a2b08c6d970a45709f099af55d84f522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.413589 1455 raft_consensus.cc:491] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.413718 1455 raft_consensus.cc:513] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.413771 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.414116 886 tablet_service.cc:1468] Processing CreateTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb (DEFAULT_TABLE table=default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.414218 1154 tablet_bootstrap.cc:654] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.414289 1455 leader_election.cc:290] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.415004 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82d58c88a6994d828b444c5814b287ff" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:44.415552 710 leader_election.cc:343] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.415894 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2bd4de753ed04021b2e3122b263bafab" } modify_external_catalogs: true
I20250629 01:51:44.416157 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0b9afb9976541b2bc7ab188f35a62ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.416252 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_afb98d5207ae4d2d8b97b8577ae3a157" } modify_external_catalogs: true
I20250629 01:51:44.421416 1158 ts_tablet_manager.cc:1939] T bd5b835ff8624ebf8956eeb965ce5950 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.424566 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347fa6a12a744ee88058643b7143e39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.425180 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648b0a0f742b4e7da9eaff6bd8795fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.425365 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c1cb87a8cbbd4ab5b62e65e5c31e2c95" } new_table_name: "default.Table_8bf46b174da647d2b1135f050ce5e430" modify_external_catalogs: true
I20250629 01:51:44.425740 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f1a2e572d6ab48b8a6f6a2b818324c38" } new_table_name: "default.Table_0685872de4674119b2386ac286878d35" modify_external_catalogs: true
I20250629 01:51:44.426030 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d5a40734fce446ae9ad039d5928a7977" } modify_external_catalogs: true
I20250629 01:51:44.426378 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95247cc778504f4d85d9013f84c59e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.427016 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54f38ccc485043ba8b8a6f644a3e4f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.428001 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6a012ba891949538800581436ef5380. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.428642 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f831a6dd68493382d5f2920d0794a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.429303 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c799a04c0f74492b9fb674f9b0da240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.430178 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a880c9bb4b94d3b9b12f28d48565f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.436738 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a47465ab39a14717b174e77b563b4100" } new_table_name: "default.Table_875984d711bd4f7da73063fdc6fba264" modify_external_catalogs: true
I20250629 01:51:44.438285 1157 tablet_replica.cc:331] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:44.441905 1540 raft_consensus.cc:491] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.442030 1540 raft_consensus.cc:513] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.442668 1540 leader_election.cc:290] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:44.444643 1159 raft_consensus.cc:383] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.444753 1159 raft_consensus.cc:738] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:51:44.444739 986 leader_election.cc:343] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:44.446164 1159 consensus_queue.cc:260] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.446734 1159 ts_tablet_manager.cc:1428] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20250629 01:51:44.449371 1159 tablet_bootstrap.cc:492] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:44.449455 1430 raft_consensus.cc:491] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.449605 1430 raft_consensus.cc:513] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.452031 1430 leader_election.cc:290] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:44.464156 1154 tablet_bootstrap.cc:492] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.464977 1154 ts_tablet_manager.cc:1397] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.179s user 0.004s sys 0.006s
I20250629 01:51:44.470008 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82d58c88a6994d828b444c5814b287ff" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:44.470216 916 raft_consensus.cc:2466] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:44.470556 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad265c77d0047c38430242c53182e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.470911 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 347fa6a12a744ee88058643b7143e39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.471485 712 leader_election.cc:304] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:44.473487 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2bd4de753ed04021b2e3122b263bafab" } modify_external_catalogs: true
I20250629 01:51:44.474447 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_afb98d5207ae4d2d8b97b8577ae3a157" } modify_external_catalogs: true
I20250629 01:51:44.475404 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d5a40734fce446ae9ad039d5928a7977" } modify_external_catalogs: true
I20250629 01:51:44.475931 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 2ad265c77d0047c38430242c53182e01 (DEFAULT_TABLE table=default.Table_ea8b3d1cbc0f46d6acfe2b800a5290a3 [id=7425128540f048519c623976fd1aaaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.476262 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad265c77d0047c38430242c53182e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.479094 1154 raft_consensus.cc:357] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.479511 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.479678 1157 raft_consensus.cc:2241] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.479840 1157 raft_consensus.cc:2270] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.480609 1154 raft_consensus.cc:383] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.480893 1154 raft_consensus.cc:738] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.481262 1157 ts_tablet_manager.cc:1905] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:44.482821 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8dd084685cc0407da9e12b0a4df386e9 consensus state has no leader
I20250629 01:51:44.483292 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_ab9e746444a24266aca5439c51b30a27 [id=60094a0a7c134476a2a4e44d8555c5d6] alter complete (version 5)
I20250629 01:51:44.483758 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 648b0a0f742b4e7da9eaff6bd8795fc4
W20250629 01:51:44.484022 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2aca26af31ed4bc994f4d62603edc881 consensus state has no leader
I20250629 01:51:44.481477 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.484988 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d456410cd09845dabd979c123f70741b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.481521 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet caf5826b7e494afc9ae884fe7e94353a with delete_type TABLET_DATA_DELETED (replaced by tablet 81b32bac5a2144d1943285f5f446d53d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.481556 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 (DEFAULT_TABLE table=default.Table_380dba3955bd458caa9a65e48c00f407 [id=77b6e244496149c5a3e46fdfb5bbe791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.485546 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20586ba2af9e4cb7adec5ca6c3b76e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.481235 1158 tablet_replica.cc:331] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.492063 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20586ba2af9e4cb7adec5ca6c3b76e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.493786 1158 raft_consensus.cc:2241] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.495057 1023 tablet_service.cc:1468] Processing CreateTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb (DEFAULT_TABLE table=default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.495110 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3db333c60cb3485a9daae316b9f1a8c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3db333c60cb3485a9daae316b9f1a8c8 already in progress: creating tablet
W20250629 01:51:44.495308 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a6a012ba891949538800581436ef5380 because the tablet was not found. No further retry: Not found: Tablet not found: a6a012ba891949538800581436ef5380
I20250629 01:51:44.495365 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7314fbdfc74b1c8d9d1be9c05244cb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.495471 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8bcc3f2a76bc4bae8b306b54e75656cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bcc3f2a76bc4bae8b306b54e75656cc already in progress: creating tablet
I20250629 01:51:44.495834 1455 raft_consensus.cc:2802] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:44.484074 891 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4320f615464fedbcab25137952f684 (DEFAULT_TABLE table=default.Table_74ed2fc8d5de4c46bbce4736df36a2be [id=5a75eb0020fb4d4fb1d2066ed7d670a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.496475 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet cade2a7984a3413a9690af48b8c28e84 (table default.Table_cdb1b42c2c774292b4ee2e323ddee5f6 [id=111d4dc551ed44a1a99a4dfcb0916eb5]) successfully deleted
I20250629 01:51:44.499128 1455 raft_consensus.cc:491] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:44.499225 1455 raft_consensus.cc:3058] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:44.500335 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4320f615464fedbcab25137952f684 (DEFAULT_TABLE table=default.Table_74ed2fc8d5de4c46bbce4736df36a2be [id=5a75eb0020fb4d4fb1d2066ed7d670a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.500658 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4320f615464fedbcab25137952f684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.504585 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.504945 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 82a837932f14419c8b32fcd2c9358215 (DEFAULT_TABLE table=default.Table_401301df62574f26a5ca95acde130d41 [id=ed3c8fa707924f8ea53d590a7f9b3d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.505218 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82a837932f14419c8b32fcd2c9358215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.509603 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet bd5b835ff8624ebf8956eeb965ce5950 (table default.Table_e3f08876e6774162a6aff102cc0e7bc9 [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
W20250629 01:51:44.509917 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0b7a640451934bff8672cd88dc86c529 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7a640451934bff8672cd88dc86c529 already in progress: creating tablet
W20250629 01:51:44.510098 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3db333c60cb3485a9daae316b9f1a8c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3db333c60cb3485a9daae316b9f1a8c8 already in progress: creating tablet
W20250629 01:51:44.510380 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2562570cb0846e19ebc97bdb43bd70d consensus state has no leader
I20250629 01:51:44.510697 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_386bd2d6e963443cb19c211c4e6e0120" } modify_external_catalogs: true
I20250629 01:51:44.510850 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_386bd2d6e963443cb19c211c4e6e0120" } modify_external_catalogs: true
I20250629 01:51:44.511848 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8e4f4f0655b44c68953073e4f0298fb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.512351 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e4f4f0655b44c68953073e4f0298fb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.512897 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_31aef3786c3841c096a18ca7abfa40df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.513151 1455 raft_consensus.cc:513] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:44.513329 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31aef3786c3841c096a18ca7abfa40df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.514076 1455 leader_election.cc:290] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.515108 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82d58c88a6994d828b444c5814b287ff" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:44.515220 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7314fbdfc74b1c8d9d1be9c05244cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.515691 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet dfee69f9484046d8abe10d2f07033e09
I20250629 01:51:44.516270 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_913feb22c480457e96599a6814b30c60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.516641 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_913feb22c480457e96599a6814b30c60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.516851 748 tablet_service.cc:1468] Processing CreateTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb (DEFAULT_TABLE table=default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.517153 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7314fbdfc74b1c8d9d1be9c05244cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.518234 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.518672 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.520799 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82d58c88a6994d828b444c5814b287ff" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:44.520958 916 raft_consensus.cc:3058] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:44.503105 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f128352ab4bd4f258d90d9a4022577a8" } new_table_name: "default.Table_1b37a535cd2f483a8c72aa10882a4301" modify_external_catalogs: true
I20250629 01:51:44.503183 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.522949 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_75f56cc1109c41f2a0a3ba83902b88ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.523054 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f
I20250629 01:51:44.503805 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2e69d514f50649f79824a1cb1da60444"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.505961 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 942f7bbe29f94083b2494aab8fbcc0c5
W20250629 01:51:44.523095 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:44.523396 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e69d514f50649f79824a1cb1da60444 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.503499 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4b91bc6605e44d8ea93074772ff7afa8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.524259 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4320f615464fedbcab25137952f684. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.524466 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b91bc6605e44d8ea93074772ff7afa8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:44.524539 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75f56cc1109c41f2a0a3ba83902b88ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.528159 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.529088 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 2249d5cba1544653b088312e7d74802e (DEFAULT_TABLE table=default.Table_338563bee1704636a7021bf819690974 [id=a2b08c6d970a45709f099af55d84f522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.529490 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2249d5cba1544653b088312e7d74802e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.529784 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c799a04c0f74492b9fb674f9b0da240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.533968 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 9cfff25bea764e62954e49611b71703c (DEFAULT_TABLE table=default.Table_7a3f650c91894a0090838ca98207ee9d [id=875222c495154458ae2052f4ca5d220f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.534341 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cfff25bea764e62954e49611b71703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.539053 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cfff25bea764e62954e49611b71703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.542436 916 raft_consensus.cc:2466] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:44.544413 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2249d5cba1544653b088312e7d74802e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.545111 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_380dba3955bd458caa9a65e48c00f407" } modify_external_catalogs: true
I20250629 01:51:44.545284 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_380dba3955bd458caa9a65e48c00f407" } modify_external_catalogs: true
W20250629 01:51:44.545778 710 leader_election.cc:343] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.548081 712 leader_election.cc:304] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:44.548203 1158 raft_consensus.cc:2270] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.548308 742 tablet_service.cc:1468] Processing CreateTablet for tablet 9cfff25bea764e62954e49611b71703c (DEFAULT_TABLE table=default.Table_7a3f650c91894a0090838ca98207ee9d [id=875222c495154458ae2052f4ca5d220f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.493366 1154 consensus_queue.cc:260] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.549697 1154 ts_tablet_manager.cc:1428] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.084s user 0.000s sys 0.003s
I20250629 01:51:44.550002 1154 tablet_bootstrap.cc:492] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:44.550846 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6d21dfd48ea4a95a2fcecde6735974b consensus state has no leader
I20250629 01:51:44.551385 755 tablet_service.cc:1468] Processing CreateTablet for tablet 2249d5cba1544653b088312e7d74802e (DEFAULT_TABLE table=default.Table_338563bee1704636a7021bf819690974 [id=a2b08c6d970a45709f099af55d84f522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.552452 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.552798 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.553001 740 tablet_service.cc:1468] Processing CreateTablet for tablet 82a837932f14419c8b32fcd2c9358215 (DEFAULT_TABLE table=default.Table_401301df62574f26a5ca95acde130d41 [id=ed3c8fa707924f8ea53d590a7f9b3d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.553136 1154 tablet_bootstrap.cc:654] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.553494 738 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4320f615464fedbcab25137952f684 (DEFAULT_TABLE table=default.Table_74ed2fc8d5de4c46bbce4736df36a2be [id=5a75eb0020fb4d4fb1d2066ed7d670a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.553318 757 tablet_service.cc:1468] Processing CreateTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 (DEFAULT_TABLE table=default.Table_380dba3955bd458caa9a65e48c00f407 [id=77b6e244496149c5a3e46fdfb5bbe791]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.554203 1158 ts_tablet_manager.cc:1905] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.559080 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82a837932f14419c8b32fcd2c9358215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.559885 1158 ts_tablet_manager.cc:1918] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.560009 1158 log.cc:1199] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:44.561046 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cfff25bea764e62954e49611b71703c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.561520 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82a837932f14419c8b32fcd2c9358215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.562057 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2249d5cba1544653b088312e7d74802e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.563997 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20586ba2af9e4cb7adec5ca6c3b76e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.564206 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4320f615464fedbcab25137952f684. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.565201 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 54162.6us
I20250629 01:51:44.566478 1455 raft_consensus.cc:2802] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:44.566735 1455 raft_consensus.cc:695] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:44.568157 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet dfee69f9484046d8abe10d2f07033e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.568508 1455 consensus_queue.cc:237] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.568807 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.569079 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.569242 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:44.571867 440 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer f304077528b34c61857a659e5873631d (127.0.104.60:32789): Couldn't send request to peer f304077528b34c61857a659e5873631d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:32789: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250629 01:51:44.571844 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet fca257d6cb614110948e1045dee44bdc with delete_type TABLET_DATA_DELETED (replaced by tablet a6a012ba891949538800581436ef5380) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.573356 1154 tablet_bootstrap.cc:492] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.573487 1154 ts_tablet_manager.cc:1397] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20250629 01:51:44.574232 1154 raft_consensus.cc:357] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.574445 1154 raft_consensus.cc:383] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.574523 1154 raft_consensus.cc:738] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.574733 1154 consensus_queue.cc:260] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.575383 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2ad265c77d0047c38430242c53182e01 (DEFAULT_TABLE table=default.Table_ea8b3d1cbc0f46d6acfe2b800a5290a3 [id=7425128540f048519c623976fd1aaaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.577008 1154 ts_tablet_manager.cc:1428] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:44.577306 1154 tablet_bootstrap.cc:492] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.577842 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad265c77d0047c38430242c53182e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.577927 1155 tablet_bootstrap.cc:492] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:44.578727 1155 ts_tablet_manager.cc:1397] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.398s user 0.010s sys 0.033s
I20250629 01:51:44.580317 1158 ts_tablet_manager.cc:1939] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.580374 741 tablet_service.cc:1515] Processing DeleteTablet for tablet dfee69f9484046d8abe10d2f07033e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.583766 1159 tablet_bootstrap.cc:654] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.586264 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet b939a8568ac14c339258af925fcb809b with delete_type TABLET_DATA_DELETED (replaced by tablet 34172d05ae40417494cd8f67f4cb981e) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.587386 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.590332 1158 tablet_replica.cc:331] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.590987 1157 ts_tablet_manager.cc:1918] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.591027 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.591302 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 02d072bf72044dc48d8090660ecb7d47 (table default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]) successfully deleted
I20250629 01:51:44.591514 1154 tablet_bootstrap.cc:654] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:44.591614 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a6a012ba891949538800581436ef5380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6a012ba891949538800581436ef5380 already in progress: creating tablet
W20250629 01:51:44.591912 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8bcc3f2a76bc4bae8b306b54e75656cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bcc3f2a76bc4bae8b306b54e75656cc already in progress: creating tablet
I20250629 01:51:44.590540 1158 raft_consensus.cc:2241] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.592406 1158 raft_consensus.cc:2270] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.592870 1158 ts_tablet_manager.cc:1905] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.597447 1013 tablet_service.cc:1468] Processing CreateTablet for tablet daf312e60c014fe698d3ec9f1a1b769c (DEFAULT_TABLE table=default.Table_d34c7a4da774448f8a7e20900b74c956 [id=df4a1725208848efa85ea19a1d3da531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.597781 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 258994dd24804f4aaa5e299436fac9b0 (DEFAULT_TABLE table=default.Table_9b58ddddb14343dcb197eff4c517689c [id=c4001552f202478e92afae5c3aacfa9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.597776 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf312e60c014fe698d3ec9f1a1b769c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.598218 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258994dd24804f4aaa5e299436fac9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.598217 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b (DEFAULT_TABLE table=default.Table_03045b3c97554436b5b29949fe6a2ccb [id=df90a81df72e4bc09675d9b0203888fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.598766 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.608179 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 4e354fce951943d69e4c666897f5caac from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:44.614658 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 20586ba2af9e4cb7adec5ca6c3b76e42
I20250629 01:51:44.615981 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.617861 743 tablet_service.cc:1468] Processing CreateTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b (DEFAULT_TABLE table=default.Table_03045b3c97554436b5b29949fe6a2ccb [id=df90a81df72e4bc09675d9b0203888fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.618089 1465 raft_consensus.cc:491] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.619349 1465 raft_consensus.cc:513] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.620975 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4e354fce951943d69e4c666897f5caac
I20250629 01:51:44.621011 1154 tablet_bootstrap.cc:492] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.621124 1154 ts_tablet_manager.cc:1397] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.001s
I20250629 01:51:44.621371 451 catalog_manager.cc:6652] ReplaceTablet: tablet 4e354fce951943d69e4c666897f5caac deleted and replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae
I20250629 01:51:44.621737 1154 raft_consensus.cc:357] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.621901 1154 raft_consensus.cc:383] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.621969 1154 raft_consensus.cc:738] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.622421 1157 log.cc:1199] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/200871d1dde144eeb59cc271ce9e4cfe
I20250629 01:51:44.622393 1387 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:44.623445 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e354fce951943d69e4c666897f5caac with delete_type TABLET_DATA_DELETED (replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.623685 1154 consensus_queue.cc:260] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.624152 1154 ts_tablet_manager.cc:1428] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:44.624398 1154 tablet_bootstrap.cc:492] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.629516 1158 ts_tablet_manager.cc:1918] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250629 01:51:44.632117 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 2 got 0
I20250629 01:51:44.635216 1157 ts_tablet_manager.cc:1939] T 200871d1dde144eeb59cc271ce9e4cfe P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:44.635900 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 2 got 1
I20250629 01:51:44.637024 1158 log.cc:1199] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/88031f8d8cbc44e4b8c53d9cd5360552
I20250629 01:51:44.637375 1158 ts_tablet_manager.cc:1939] T 88031f8d8cbc44e4b8c53d9cd5360552 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.637593 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 2 got 0
I20250629 01:51:44.638270 1158 tablet_replica.cc:331] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.638449 1158 raft_consensus.cc:2241] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.638448 1155 raft_consensus.cc:357] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.638592 1158 raft_consensus.cc:2270] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.630292 1154 tablet_bootstrap.cc:654] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.638962 1465 leader_election.cc:290] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.639442 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 200871d1dde144eeb59cc271ce9e4cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.639777 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fca257d6cb614110948e1045dee44bdc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:44.639971 1050 raft_consensus.cc:2466] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:44.640584 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_328919206d1e44838bc19f8a2ff9d371 [id=ac80888442774af495a53bba44dfa763]) successfully deleted
W20250629 01:51:44.640825 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 75ce1bd4af434f7e99092aafc0ab0118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ce1bd4af434f7e99092aafc0ab0118 already in progress: creating tablet
I20250629 01:51:44.641490 754 tablet_service.cc:1468] Processing CreateTablet for tablet daf312e60c014fe698d3ec9f1a1b769c (DEFAULT_TABLE table=default.Table_d34c7a4da774448f8a7e20900b74c956 [id=df4a1725208848efa85ea19a1d3da531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.641824 1157 tablet_replica.cc:331] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:44.641988 1157 raft_consensus.cc:2241] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.642141 1157 raft_consensus.cc:2270] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.642323 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.642521 1364 raft_consensus.cc:491] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.642560 1157 ts_tablet_manager.cc:1905] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.642647 1364 raft_consensus.cc:513] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.643030 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fca257d6cb614110948e1045dee44bdc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:44.643102 1364 leader_election.cc:290] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:44.644224 1158 ts_tablet_manager.cc:1905] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.645763 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:44.646443 710 leader_election.cc:304] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:44.648532 1154 tablet_bootstrap.cc:492] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.648665 1154 ts_tablet_manager.cc:1397] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250629 01:51:44.649240 753 tablet_service.cc:1468] Processing CreateTablet for tablet 258994dd24804f4aaa5e299436fac9b0 (DEFAULT_TABLE table=default.Table_9b58ddddb14343dcb197eff4c517689c [id=c4001552f202478e92afae5c3aacfa9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.649441 1154 raft_consensus.cc:357] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.649753 1154 raft_consensus.cc:383] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.652611 1382 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:44.656147 1154 raft_consensus.cc:738] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.656347 1154 consensus_queue.cc:260] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.657541 1154 ts_tablet_manager.cc:1428] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250629 01:51:44.657781 1154 tablet_bootstrap.cc:492] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.659473 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 582e063f97ab415cab126845789e6e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.660847 1154 tablet_bootstrap.cc:654] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.661674 916 raft_consensus.cc:1273] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:44.662705 1158 ts_tablet_manager.cc:1918] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.662817 1158 log.cc:1199] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/caf5826b7e494afc9ae884fe7e94353a
I20250629 01:51:44.663115 1158 ts_tablet_manager.cc:1939] T caf5826b7e494afc9ae884fe7e94353a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.663882 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet caf5826b7e494afc9ae884fe7e94353a (table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]) successfully deleted
I20250629 01:51:44.664505 1158 tablet_replica.cc:331] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.664657 1158 raft_consensus.cc:2241] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.664805 1158 raft_consensus.cc:2270] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:44.665067 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f already in progress: creating tablet
I20250629 01:51:44.665238 1158 ts_tablet_manager.cc:1905] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:44.665238 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648b0a0f742b4e7da9eaff6bd8795fc4 already in progress: creating tablet
W20250629 01:51:44.665410 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet dfee69f9484046d8abe10d2f07033e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfee69f9484046d8abe10d2f07033e09 already in progress: creating tablet
W20250629 01:51:44.665601 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 942f7bbe29f94083b2494aab8fbcc0c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 942f7bbe29f94083b2494aab8fbcc0c5 already in progress: creating tablet
I20250629 01:51:44.666745 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 02d072bf72044dc48d8090660ecb7d47 (table default.Table_2d68cf4935ca467fa49eb226d95d067a [id=7e7f80a628554a8395203c70311156ab]) successfully deleted
I20250629 01:51:44.667631 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 200871d1dde144eeb59cc271ce9e4cfe (table default.Table_3ac2dc2474da47f7ba25608e622e9a68 [id=5b172406f58f4023b2b76511690541f6]) successfully deleted
W20250629 01:51:44.667841 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e962a020664b42ea918e5952dc43aade because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e962a020664b42ea918e5952dc43aade already in progress: creating tablet
W20250629 01:51:44.668008 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f5941dd6381041538be6c92858d6ba13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5941dd6381041538be6c92858d6ba13 already in progress: creating tablet
I20250629 01:51:44.668381 988 leader_election.cc:304] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:44.668610 988 raft_consensus.cc:2684] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250629 01:51:44.669021 988 leader_election.cc:343] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: 02d072bf72044dc48d8090660ecb7d47
I20250629 01:51:44.669162 988 leader_election.cc:304] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:44.669313 988 raft_consensus.cc:2684] T 02d072bf72044dc48d8090660ecb7d47 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:44.669816 1154 tablet_bootstrap.cc:492] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.669991 1154 ts_tablet_manager.cc:1397] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20250629 01:51:44.669960 1364 consensus_queue.cc:1035] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:44.670550 1154 raft_consensus.cc:357] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.670710 1154 raft_consensus.cc:383] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.670778 1154 raft_consensus.cc:738] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.671089 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f98bb48081984d23af55d7fa6a8bec7b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:44.671223 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "575006c12f7c41658e234f7c6c425d69" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:44.671289 1154 consensus_queue.cc:260] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.671515 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648fd1ca8f514ea3ba4554452b19933f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:44.671680 1154 ts_tablet_manager.cc:1428] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250629 01:51:44.669816 988 leader_election.cc:343] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: b939a8568ac14c339258af925fcb809b
I20250629 01:51:44.673126 1154 tablet_bootstrap.cc:492] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.674448 1154 tablet_bootstrap.cc:654] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.676631 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 with delete_type TABLET_DATA_DELETED (replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.679824 751 tablet_service.cc:1468] Processing CreateTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a (DEFAULT_TABLE table=default.Table_75f56cc1109c41f2a0a3ba83902b88ea [id=fcdd225098994f2f8c425a8fc3165d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.680955 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a (DEFAULT_TABLE table=default.Table_75f56cc1109c41f2a0a3ba83902b88ea [id=fcdd225098994f2f8c425a8fc3165d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.681278 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b3badd684d45fa88dc8cbcfc61001a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.684186 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 06e31520d3ad4c3395a59049755a4725 (DEFAULT_TABLE table=default.Table_2e69d514f50649f79824a1cb1da60444 [id=d843c94cf5f847249d82ea7944980d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.684489 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e31520d3ad4c3395a59049755a4725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.684437 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3d334b0f295e42378d74e5e746d911e2" } new_table_name: "default.Table_d7789b77f4814c388e1eb092297120b1" modify_external_catalogs: true
I20250629 01:51:44.687561 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d6363385f77141019261e5492e744cbd" } new_table_name: "default.Table_8ace6d33743f4b9aa9a7ec9e11f5a0b6" modify_external_catalogs: true
I20250629 01:51:44.688982 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e3b94f8250a143b096703f2b0386e3cd (DEFAULT_TABLE table=default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.689265 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b94f8250a143b096703f2b0386e3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.689770 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6b7f597bfe3947f884cf5e129049c99b" } modify_external_catalogs: true
I20250629 01:51:44.689913 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6b7f597bfe3947f884cf5e129049c99b" } modify_external_catalogs: true
I20250629 01:51:44.693045 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f6acf9f470204fdb876176dc5c191895"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.693269 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 3ba0e267f48a4bf1b58fb2339694719f from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:44.693528 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6acf9f470204fdb876176dc5c191895 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:44.696151 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fcead51fd9b43d0a52c4669a5e23fe5 consensus state has no leader
I20250629 01:51:44.700639 1158 ts_tablet_manager.cc:1918] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.700745 1158 log.cc:1199] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9c7f1f2485cb4bf3b8e0dbdd60272b7f
I20250629 01:51:44.701081 1158 ts_tablet_manager.cc:1939] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.701884 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]) successfully deleted
W20250629 01:51:44.702119 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 consensus state has no leader
I20250629 01:51:44.702811 1158 tablet_replica.cc:331] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.703857 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fbcae02d893f4795a12d26af57b18b85 (DEFAULT_TABLE table=default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 [id=a128a46807814758b9b79071ca6db923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.704209 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcae02d893f4795a12d26af57b18b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.704699 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 5e9f6de505a34afbae48a373b4b2a0cf from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:44.704912 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.705716 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.706281 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:44.706403 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b (DEFAULT_TABLE table=default.Table_31aef3786c3841c096a18ca7abfa40df [id=b46cf5dd7d524d0cb23f236b318eafca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.706728 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.707759 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cedbd093b4420d8f7b29219ce8f77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.708288 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet e83009c1599a43d18d1c73c0101ab0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.706960 895 tablet_service.cc:1515] Processing DeleteTablet for tablet dfee69f9484046d8abe10d2f07033e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.710271 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ac9d77381f7e4241aa546eaea3077168 (DEFAULT_TABLE table=default.Table_4b91bc6605e44d8ea93074772ff7afa8 [id=ceaecddcc74f48b4ad47d03ea91dd9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.710567 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9d77381f7e4241aa546eaea3077168. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.710659 749 tablet_service.cc:1515] Processing DeleteTablet for tablet e83009c1599a43d18d1c73c0101ab0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.713049 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 6092e19acf974f2dad5a903da72944ba (DEFAULT_TABLE table=default.Table_913feb22c480457e96599a6814b30c60 [id=abbf099fb0aa4b118f76e5c48700df39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.713397 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6092e19acf974f2dad5a903da72944ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.716176 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a (DEFAULT_TABLE table=default.Table_f2175d9af5064903ab1c97c2035e2527 [id=207fae1412fa4f1da7eb67dadd6f986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.716348 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 40592f11f430469f9c8e82eba25b38ed (DEFAULT_TABLE table=default.Table_8e4f4f0655b44c68953073e4f0298fb8 [id=c0e46a0b48ee4ae58b819cd01a157a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.716487 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9ef77181ba48f3a6de16b945c8df1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.716697 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40592f11f430469f9c8e82eba25b38ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.719484 880 tablet_service.cc:1468] Processing CreateTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b (DEFAULT_TABLE table=default.Table_03045b3c97554436b5b29949fe6a2ccb [id=df90a81df72e4bc09675d9b0203888fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.720137 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e39f46841d454041bb064e74d3c27bc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.720564 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e39f46841d454041bb064e74d3c27bc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.721239 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55354f7b2e844ba292e478667d1f0d76" } new_table_name: "default.Table_5f6743b197734f50999a12d6864e1134" modify_external_catalogs: true
I20250629 01:51:44.722301 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_59f54bc37cc14d9fbccad27fdbc6e62e" } new_table_name: "default.Table_047550de222e41deaa74e50cd86bdea8" modify_external_catalogs: true
I20250629 01:51:44.723289 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_31aef3786c3841c096a18ca7abfa40df" } modify_external_catalogs: true
I20250629 01:51:44.723450 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_31aef3786c3841c096a18ca7abfa40df" } modify_external_catalogs: true
I20250629 01:51:44.724560 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7acc3d53b2754aef87214be4a094b1f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.724988 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7acc3d53b2754aef87214be4a094b1f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.726037 745 tablet_service.cc:1468] Processing CreateTablet for tablet 40592f11f430469f9c8e82eba25b38ed (DEFAULT_TABLE table=default.Table_8e4f4f0655b44c68953073e4f0298fb8 [id=c0e46a0b48ee4ae58b819cd01a157a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.726207 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_10a0ae1fcff049dda2c11881c3191fa3" } modify_external_catalogs: true
I20250629 01:51:44.726361 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_10a0ae1fcff049dda2c11881c3191fa3" } modify_external_catalogs: true
I20250629 01:51:44.726567 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f220872b43b3433694c2f522f6e54acd" } modify_external_catalogs: true
I20250629 01:51:44.726850 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f220872b43b3433694c2f522f6e54acd" } modify_external_catalogs: true
I20250629 01:51:44.726884 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (DEFAULT_TABLE table=default.Table_8ace6d33743f4b9aa9a7ec9e11f5a0b6 [id=13e374d69c0d467282f126d666c25317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.727279 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcead51fd9b43d0a52c4669a5e23fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.728892 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e354fce951943d69e4c666897f5caac with delete_type TABLET_DATA_DELETED (replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.729573 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.730566 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d532629a716e44368b4324cf8b41f873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.730926 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d532629a716e44368b4324cf8b41f873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.733069 886 tablet_service.cc:1468] Processing CreateTablet for tablet 258994dd24804f4aaa5e299436fac9b0 (DEFAULT_TABLE table=default.Table_9b58ddddb14343dcb197eff4c517689c [id=c4001552f202478e92afae5c3aacfa9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.733393 895 tablet_service.cc:1468] Processing CreateTablet for tablet daf312e60c014fe698d3ec9f1a1b769c (DEFAULT_TABLE table=default.Table_d34c7a4da774448f8a7e20900b74c956 [id=df4a1725208848efa85ea19a1d3da531]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.735137 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 798516e55ca747fcb2942026adb78af0 consensus state has no leader
I20250629 01:51:44.702994 1158 raft_consensus.cc:2241] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.735741 1158 raft_consensus.cc:2270] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.735786 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.736343 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 consensus state has no leader
I20250629 01:51:44.736603 1154 tablet_bootstrap.cc:492] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.736721 1154 ts_tablet_manager.cc:1397] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.002s
I20250629 01:51:44.737459 1154 raft_consensus.cc:357] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.737684 1154 raft_consensus.cc:383] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.737761 1154 raft_consensus.cc:738] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.737768 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.737933 1465 raft_consensus.cc:2802] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:44.738005 1465 raft_consensus.cc:491] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:44.737962 1154 consensus_queue.cc:260] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.738085 1465 raft_consensus.cc:3058] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:44.739284 1154 ts_tablet_manager.cc:1428] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:44.739650 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258994dd24804f4aaa5e299436fac9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.739863 1154 tablet_bootstrap.cc:492] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:44.740258 712 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: Tablet not RUNNING: STOPPED
W20250629 01:51:44.740562 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 97062us
I20250629 01:51:44.741432 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "575006c12f7c41658e234f7c6c425d69" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:44.741715 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648fd1ca8f514ea3ba4554452b19933f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:44.741715 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f98bb48081984d23af55d7fa6a8bec7b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:44.742339 1465 raft_consensus.cc:513] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.742424 1158 ts_tablet_manager.cc:1905] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.742501 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf312e60c014fe698d3ec9f1a1b769c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.742895 1465 leader_election.cc:290] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.743472 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fca257d6cb614110948e1045dee44bdc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
W20250629 01:51:44.743929 710 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:44.744382 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e354fce951943d69e4c666897f5caac with delete_type TABLET_DATA_DELETED (replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.744590 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf312e60c014fe698d3ec9f1a1b769c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.745041 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b3badd684d45fa88dc8cbcfc61001a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.748159 1154 tablet_bootstrap.cc:654] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.748224 747 tablet_service.cc:1468] Processing CreateTablet for tablet ac9d77381f7e4241aa546eaea3077168 (DEFAULT_TABLE table=default.Table_4b91bc6605e44d8ea93074772ff7afa8 [id=ceaecddcc74f48b4ad47d03ea91dd9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.748515 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9d77381f7e4241aa546eaea3077168. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.748726 890 tablet_service.cc:1468] Processing CreateTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a (DEFAULT_TABLE table=default.Table_75f56cc1109c41f2a0a3ba83902b88ea [id=fcdd225098994f2f8c425a8fc3165d24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.749009 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b3badd684d45fa88dc8cbcfc61001a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.750288 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fca257d6cb614110948e1045dee44bdc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:44.750769 883 tablet_service.cc:1468] Processing CreateTablet for tablet e3b94f8250a143b096703f2b0386e3cd (DEFAULT_TABLE table=default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.751029 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b94f8250a143b096703f2b0386e3cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.751003 986 leader_election.cc:343] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: b939a8568ac14c339258af925fcb809b
I20250629 01:51:44.751114 986 leader_election.cc:304] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:44.751485 1364 raft_consensus.cc:2747] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:44.751796 712 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:44.751891 712 leader_election.cc:304] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:44.752519 1156 ts_tablet_manager.cc:1918] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250629 01:51:44.752666 1156 log.cc:1199] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/88031f8d8cbc44e4b8c53d9cd5360552
I20250629 01:51:44.753051 1156 ts_tablet_manager.cc:1939] T 88031f8d8cbc44e4b8c53d9cd5360552 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.753186 1387 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:44.753824 1158 ts_tablet_manager.cc:1918] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.753948 1158 log.cc:1199] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/fca257d6cb614110948e1045dee44bdc
I20250629 01:51:44.754139 885 tablet_service.cc:1468] Processing CreateTablet for tablet 06e31520d3ad4c3395a59049755a4725 (DEFAULT_TABLE table=default.Table_2e69d514f50649f79824a1cb1da60444 [id=d843c94cf5f847249d82ea7944980d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.754274 1158 ts_tablet_manager.cc:1939] T fca257d6cb614110948e1045dee44bdc P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.754428 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e31520d3ad4c3395a59049755a4725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.752286 1465 raft_consensus.cc:2747] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250629 01:51:44.755098 1158 tablet_replica.cc:331] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.755126 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet fca257d6cb614110948e1045dee44bdc (table default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]) successfully deleted
I20250629 01:51:44.755249 1158 raft_consensus.cc:2241] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.755390 1158 raft_consensus.cc:2270] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.757323 1156 tablet_replica.cc:331] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:44.757443 739 tablet_service.cc:1468] Processing CreateTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b (DEFAULT_TABLE table=default.Table_31aef3786c3841c096a18ca7abfa40df [id=b46cf5dd7d524d0cb23f236b318eafca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.759635 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cedbd093b4420d8f7b29219ce8f77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.759708 878 tablet_service.cc:1468] Processing CreateTablet for tablet fbcae02d893f4795a12d26af57b18b85 (DEFAULT_TABLE table=default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 [id=a128a46807814758b9b79071ca6db923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.759974 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcae02d893f4795a12d26af57b18b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.760324 1154 tablet_bootstrap.cc:492] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:44.760442 1154 ts_tablet_manager.cc:1397] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20250629 01:51:44.761054 1154 raft_consensus.cc:357] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.761214 1154 raft_consensus.cc:383] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.761278 1154 raft_consensus.cc:738] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.761487 1154 consensus_queue.cc:260] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.762048 1154 ts_tablet_manager.cc:1428] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:44.762473 1154 tablet_bootstrap.cc:492] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.764737 1547 raft_consensus.cc:491] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.764801 1158 ts_tablet_manager.cc:1905] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.764861 1547 raft_consensus.cc:513] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.764919 748 tablet_service.cc:1468] Processing CreateTablet for tablet e3b94f8250a143b096703f2b0386e3cd (DEFAULT_TABLE table=default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.765198 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b94f8250a143b096703f2b0386e3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.765393 1547 leader_election.cc:290] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:44.765795 892 tablet_service.cc:1468] Processing CreateTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b (DEFAULT_TABLE table=default.Table_31aef3786c3841c096a18ca7abfa40df [id=b46cf5dd7d524d0cb23f236b318eafca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.766314 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cedbd093b4420d8f7b29219ce8f77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.768252 755 tablet_service.cc:1468] Processing CreateTablet for tablet fbcae02d893f4795a12d26af57b18b85 (DEFAULT_TABLE table=default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 [id=a128a46807814758b9b79071ca6db923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.768523 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbcae02d893f4795a12d26af57b18b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.769690 1154 tablet_bootstrap.cc:654] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.773917 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (DEFAULT_TABLE table=default.Table_8ace6d33743f4b9aa9a7ec9e11f5a0b6 [id=13e374d69c0d467282f126d666c25317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.774118 738 tablet_service.cc:1468] Processing CreateTablet for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (DEFAULT_TABLE table=default.Table_8ace6d33743f4b9aa9a7ec9e11f5a0b6 [id=13e374d69c0d467282f126d666c25317]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.774226 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcead51fd9b43d0a52c4669a5e23fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.774466 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcead51fd9b43d0a52c4669a5e23fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.776252 1158 ts_tablet_manager.cc:1918] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.776510 1158 log.cc:1199] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b939a8568ac14c339258af925fcb809b
I20250629 01:51:44.777489 1158 ts_tablet_manager.cc:1939] T b939a8568ac14c339258af925fcb809b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.777704 757 tablet_service.cc:1468] Processing CreateTablet for tablet 06e31520d3ad4c3395a59049755a4725 (DEFAULT_TABLE table=default.Table_2e69d514f50649f79824a1cb1da60444 [id=d843c94cf5f847249d82ea7944980d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.778119 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06e31520d3ad4c3395a59049755a4725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.779760 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b939a8568ac14c339258af925fcb809b (table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]) successfully deleted
W20250629 01:51:44.780297 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20586ba2af9e4cb7adec5ca6c3b76e42 already in progress: creating tablet
I20250629 01:51:44.780679 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_44f5d1913b0e4eedaac117acfdf02c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.780841 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4e354fce951943d69e4c666897f5caac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e354fce951943d69e4c666897f5caac already in progress: creating tablet
I20250629 01:51:44.781141 1158 tablet_replica.cc:331] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:44.781201 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44f5d1913b0e4eedaac117acfdf02c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.782269 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e3283d56ec2743e2a1073ee8909e103f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.782694 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3283d56ec2743e2a1073ee8909e103f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.783118 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_98a1aa99dbcb4f448201bde47757f522"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.783427 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a91740f16744439f55077c4fbadee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:44.783540 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98a1aa99dbcb4f448201bde47757f522 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.786123 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 258994dd24804f4aaa5e299436fac9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.786865 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_49d0e0f59d084d388696b7b11d6d50d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.787302 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49d0e0f59d084d388696b7b11d6d50d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.788600 889 tablet_service.cc:1468] Processing CreateTablet for tablet 40592f11f430469f9c8e82eba25b38ed (DEFAULT_TABLE table=default.Table_8e4f4f0655b44c68953073e4f0298fb8 [id=c0e46a0b48ee4ae58b819cd01a157a47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.788873 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40592f11f430469f9c8e82eba25b38ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.781688 1158 raft_consensus.cc:2241] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.789897 1158 raft_consensus.cc:2270] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.791541 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_328919206d1e44838bc19f8a2ff9d371 [id=ac80888442774af495a53bba44dfa763]) successfully deleted
W20250629 01:51:44.791795 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 75ce1bd4af434f7e99092aafc0ab0118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ce1bd4af434f7e99092aafc0ab0118 already in progress: creating tablet
W20250629 01:51:44.791965 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f already in progress: creating tablet
W20250629 01:51:44.792171 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648b0a0f742b4e7da9eaff6bd8795fc4 already in progress: creating tablet
W20250629 01:51:44.792378 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet dfee69f9484046d8abe10d2f07033e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfee69f9484046d8abe10d2f07033e09 already in progress: creating tablet
W20250629 01:51:44.792572 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 942f7bbe29f94083b2494aab8fbcc0c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 942f7bbe29f94083b2494aab8fbcc0c5 already in progress: creating tablet
I20250629 01:51:44.792853 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6092e19acf974f2dad5a903da72944ba (DEFAULT_TABLE table=default.Table_913feb22c480457e96599a6814b30c60 [id=abbf099fb0aa4b118f76e5c48700df39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.793134 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6092e19acf974f2dad5a903da72944ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.794284 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ef976703d61442f8987ff32070a33592
I20250629 01:51:44.794816 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f9284d7ccab1429d91910af2fd53b89c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.795194 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9284d7ccab1429d91910af2fd53b89c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.795229 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_836f6d2993e14d3d85372201bb2cf8f2" } new_table_name: "default.Table_58dcfb06bfe149d28ad21cc67702f6d5" modify_external_catalogs: true
I20250629 01:51:44.797011 1158 ts_tablet_manager.cc:1905] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.799664 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet ef976703d61442f8987ff32070a33592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.799723 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ef976703d61442f8987ff32070a33592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.800045 1156 raft_consensus.cc:2241] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.800274 1156 raft_consensus.cc:2270] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.800827 1156 ts_tablet_manager.cc:1905] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.801447 887 tablet_service.cc:1468] Processing CreateTablet for tablet 6092e19acf974f2dad5a903da72944ba (DEFAULT_TABLE table=default.Table_913feb22c480457e96599a6814b30c60 [id=abbf099fb0aa4b118f76e5c48700df39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.801733 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6092e19acf974f2dad5a903da72944ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.803803 1158 ts_tablet_manager.cc:1918] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.803934 1158 log.cc:1199] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/200871d1dde144eeb59cc271ce9e4cfe
I20250629 01:51:44.803944 740 tablet_service.cc:1468] Processing CreateTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a (DEFAULT_TABLE table=default.Table_f2175d9af5064903ab1c97c2035e2527 [id=207fae1412fa4f1da7eb67dadd6f986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.804245 1158 ts_tablet_manager.cc:1939] T 200871d1dde144eeb59cc271ce9e4cfe P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.804389 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9ef77181ba48f3a6de16b945c8df1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.804896 1158 tablet_replica.cc:331] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.805047 1158 raft_consensus.cc:2241] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.805178 1158 raft_consensus.cc:2270] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.808382 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 200871d1dde144eeb59cc271ce9e4cfe (table default.Table_3ac2dc2474da47f7ba25608e622e9a68 [id=5b172406f58f4023b2b76511690541f6]) successfully deleted
I20250629 01:51:44.808524 1158 ts_tablet_manager.cc:1905] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.809525 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6b2a627b815e45c480dcaafe356a5d50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:44.809939 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b2a627b815e45c480dcaafe356a5d50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.811858 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40592f11f430469f9c8e82eba25b38ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.813614 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e9f6de505a34afbae48a373b4b2a0cf" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:44.814275 1158 ts_tablet_manager.cc:1918] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.814360 1158 log.cc:1199] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/582e063f97ab415cab126845789e6e20
I20250629 01:51:44.814396 773 raft_consensus.cc:1273] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 443c4bd3c5d548ee8cd347d2b4de5293: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:44.814683 1158 ts_tablet_manager.cc:1939] T 582e063f97ab415cab126845789e6e20 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.815466 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 582e063f97ab415cab126845789e6e20 (table default.Table_30a1da646c914cde85078e3e1270a01e [id=26e8b18cac084e4c9e603950a6595c35]) successfully deleted
I20250629 01:51:44.815734 1158 tablet_replica.cc:331] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.815892 1158 raft_consensus.cc:2241] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:44.816015 1158 raft_consensus.cc:2270] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.816706 1158 ts_tablet_manager.cc:1905] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.816915 1382 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:44.818051 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_30467a20b5b54ca7894c405b22705bc2" } modify_external_catalogs: true
I20250629 01:51:44.818181 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_30467a20b5b54ca7894c405b22705bc2" } modify_external_catalogs: true
I20250629 01:51:44.818655 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40] alter complete (version 2)
I20250629 01:51:44.819020 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5e9f6de505a34afbae48a373b4b2a0cf" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:44.819183 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet fb359994b20748518a676045ac6f0df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.820570 1155 raft_consensus.cc:383] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.820652 1155 raft_consensus.cc:738] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.820889 1155 consensus_queue.cc:260] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.821422 1155 ts_tablet_manager.cc:1428] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.242s user 0.000s sys 0.003s
I20250629 01:51:44.822041 1155 tablet_bootstrap.cc:492] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:44.822394 1158 ts_tablet_manager.cc:1918] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.822512 1158 log.cc:1199] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5f3bf5fd9b674eeb8a5a029152cbae13
I20250629 01:51:44.822863 1158 ts_tablet_manager.cc:1939] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.823618 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5f3bf5fd9b674eeb8a5a029152cbae13 (table default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]) successfully deleted
I20250629 01:51:44.823734 1158 tablet_replica.cc:331] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.823879 1158 raft_consensus.cc:2241] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:44.823951 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e83009c1599a43d18d1c73c0101ab0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e83009c1599a43d18d1c73c0101ab0be already in progress: creating tablet
I20250629 01:51:44.824028 1158 raft_consensus.cc:2270] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.824331 1155 tablet_bootstrap.cc:654] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.824596 1158 ts_tablet_manager.cc:1905] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.827840 877 tablet_service.cc:1468] Processing CreateTablet for tablet ac9d77381f7e4241aa546eaea3077168 (DEFAULT_TABLE table=default.Table_4b91bc6605e44d8ea93074772ff7afa8 [id=ceaecddcc74f48b4ad47d03ea91dd9d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.828186 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9d77381f7e4241aa546eaea3077168. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.831714 1158 ts_tablet_manager.cc:1918] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.831822 1158 log.cc:1199] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d3a91740f16744439f55077c4fbadee1
I20250629 01:51:44.832201 1158 ts_tablet_manager.cc:1939] T d3a91740f16744439f55077c4fbadee1 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.832204 892 tablet_service.cc:1515] Processing DeleteTablet for tablet e83009c1599a43d18d1c73c0101ab0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.833070 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d3a91740f16744439f55077c4fbadee1 (table default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 [id=cf0d80c2c9e2464db9ded94438df584a]) successfully deleted
I20250629 01:51:44.833163 1158 tablet_replica.cc:331] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:44.833371 1158 raft_consensus.cc:2241] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:44.833387 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ef976703d61442f8987ff32070a33592 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef976703d61442f8987ff32070a33592 already in progress: creating tablet
I20250629 01:51:44.833521 1158 raft_consensus.cc:2270] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:44.834273 1158 ts_tablet_manager.cc:1905] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:44.839340 1547 consensus_queue.cc:1035] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:51:44.839907 850 leader_election.cc:343] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:44.840181 850 leader_election.cc:343] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:44.840415 850 leader_election.cc:343] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.841040 1547 consensus_queue.cc:1035] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:44.841100 1158 ts_tablet_manager.cc:1918] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.841243 1158 log.cc:1199] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/fb359994b20748518a676045ac6f0df8
I20250629 01:51:44.841589 1158 ts_tablet_manager.cc:1939] T fb359994b20748518a676045ac6f0df8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:44.841830 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a (DEFAULT_TABLE table=default.Table_f2175d9af5064903ab1c97c2035e2527 [id=207fae1412fa4f1da7eb67dadd6f986e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.842183 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9ef77181ba48f3a6de16b945c8df1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.844033 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet fb359994b20748518a676045ac6f0df8 (table default.Table_a23d7cbbf7804d2e95f76b62ec45c158 [id=4428e6c03ff541c89ad394b08004be0b]) successfully deleted
W20250629 01:51:44.845702 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26d85fd3d3dd4b07a812c019e1a5c748 consensus state has no leader
W20250629 01:51:44.848356 850 leader_election.cc:343] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:44.848773 850 leader_election.cc:343] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.848884 850 leader_election.cc:304] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:44.849470 850 consensus_peers.cc:489] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:51:44.849658 850 leader_election.cc:343] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.849786 850 leader_election.cc:304] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:44.849929 1548 raft_consensus.cc:2747] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:44.850078 1548 raft_consensus.cc:2747] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:44.850337 850 leader_election.cc:343] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.850482 850 leader_election.cc:304] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:44.850739 1430 raft_consensus.cc:2747] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:44.850950 850 leader_election.cc:343] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:44.851092 850 leader_election.cc:304] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:44.851347 1548 raft_consensus.cc:2747] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:44.851609 850 consensus_peers.cc:489] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88031f8d8cbc44e4b8c53d9cd5360552. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:44.851583 1548 consensus_queue.cc:1035] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:44.851727 1156 ts_tablet_manager.cc:1918] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.851835 1156 log.cc:1199] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9c7f1f2485cb4bf3b8e0dbdd60272b7f
I20250629 01:51:44.852944 887 tablet_service.cc:1515] Processing DeleteTablet for tablet ef976703d61442f8987ff32070a33592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.856164 454 catalog_manager.cc:5582] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:44.856534 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:44.861788 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 36fcb819c61c4fabbb0da295d0002998 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:44.862102 748 tablet_service.cc:1468] Processing CreateTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.862216 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 819ed44d98f14075b9a67b88dc44e86e (DEFAULT_TABLE table=default.Table_f6acf9f470204fdb876176dc5c191895 [id=0070a00fcaa945cfb2e47f3c3e347323]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.862278 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.862555 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819ed44d98f14075b9a67b88dc44e86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.862644 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a81e6f9711c4d398ae09f33a75566ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.862864 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a81e6f9711c4d398ae09f33a75566ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.868633 755 tablet_service.cc:1468] Processing CreateTablet for tablet 819ed44d98f14075b9a67b88dc44e86e (DEFAULT_TABLE table=default.Table_f6acf9f470204fdb876176dc5c191895 [id=0070a00fcaa945cfb2e47f3c3e347323]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.869309 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819ed44d98f14075b9a67b88dc44e86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.869729 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 05905638c6e241a095438f277f53adab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.869995 877 tablet_service.cc:1468] Processing CreateTablet for tablet 819ed44d98f14075b9a67b88dc44e86e (DEFAULT_TABLE table=default.Table_f6acf9f470204fdb876176dc5c191895 [id=0070a00fcaa945cfb2e47f3c3e347323]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.870211 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 26d85fd3d3dd4b07a812c019e1a5c748 (table default.Table_58dcfb06bfe149d28ad21cc67702f6d5 [id=b9df439e13104b14be7c1b522b04ec13], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 26d85fd3d3dd4b07a812c019e1a5c748 consensus state has no leader
I20250629 01:51:44.870335 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819ed44d98f14075b9a67b88dc44e86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.871970 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0b16b6840c24484b90218fffc1da9621
I20250629 01:51:44.872798 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b16b6840c24484b90218fffc1da9621 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.872859 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b16b6840c24484b90218fffc1da9621 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.872670 887 tablet_service.cc:1468] Processing CreateTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.874154 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a81e6f9711c4d398ae09f33a75566ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.874228 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0b16b6840c24484b90218fffc1da9621 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b16b6840c24484b90218fffc1da9621 already in progress: creating tablet
I20250629 01:51:44.884241 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 575006c12f7c41658e234f7c6c425d69 with delete_type TABLET_DATA_DELETED (replaced by tablet a792a26725954adea15ad57e31bfb224) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.886758 1155 tablet_bootstrap.cc:492] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:44.887254 1155 ts_tablet_manager.cc:1397] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20250629 01:51:44.888196 1155 raft_consensus.cc:357] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.888469 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 68cedbd093b4420d8f7b29219ce8f77b
I20250629 01:51:44.891301 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:44.892575 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 68cedbd093b4420d8f7b29219ce8f77b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68cedbd093b4420d8f7b29219ce8f77b already in progress: creating tablet
I20250629 01:51:44.893622 1339 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:44.898123 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
I20250629 01:51:44.898897 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ed13689efff54739bc1cd029699b4384" } new_table_name: "default.Table_a9833281c36943b1a5a8009f5fc437bd" modify_external_catalogs: true
I20250629 01:51:44.899544 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
I20250629 01:51:44.900019 1157 ts_tablet_manager.cc:1918] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:44.900946 1154 tablet_bootstrap.cc:492] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:51:44.901029 850 consensus_peers.cc:489] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Couldn't send request to peer ef5c5979bb9e475f983af7c3e9e0e019. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:44.901072 1154 ts_tablet_manager.cc:1397] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.139s user 0.024s sys 0.012s
I20250629 01:51:44.902341 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b16b6840c24484b90218fffc1da9621 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.902135 1154 raft_consensus.cc:357] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.902878 1154 raft_consensus.cc:383] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.902968 1154 raft_consensus.cc:738] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.903568 1155 raft_consensus.cc:383] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.903793 1155 raft_consensus.cc:738] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.904038 1155 consensus_queue.cc:260] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.904598 1155 ts_tablet_manager.cc:1428] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250629 01:51:44.891913 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.896413 1156 ts_tablet_manager.cc:1939] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:44.905331 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 2 got 0
I20250629 01:51:44.906185 1159 tablet_bootstrap.cc:492] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:44.906128 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_12d6ac1cd45c43eeac32cc51d3aae591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.906296 1159 ts_tablet_manager.cc:1397] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.459s user 0.010s sys 0.005s
I20250629 01:51:44.906502 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0072df8d087f4e8e9541d7f90a27bf4b" } new_table_name: "default.Table_5c353375f2c64ac39356062ccc4a5986" modify_external_catalogs: true
W20250629 01:51:44.906608 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12d6ac1cd45c43eeac32cc51d3aae591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.907002 1159 raft_consensus.cc:357] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.907171 1159 raft_consensus.cc:383] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:44.907258 1159 raft_consensus.cc:738] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:44.907454 1159 consensus_queue.cc:260] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:44.907863 1154 consensus_queue.cc:260] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.907980 1159 ts_tablet_manager.cc:1428] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:44.908353 1154 ts_tablet_manager.cc:1428] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:44.908576 1154 tablet_bootstrap.cc:492] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:44.910545 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_875984d711bd4f7da73063fdc6fba264" } modify_external_catalogs: true
I20250629 01:51:44.910710 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_875984d711bd4f7da73063fdc6fba264" } modify_external_catalogs: true
I20250629 01:51:44.911032 1155 tablet_bootstrap.cc:492] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:44.912822 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b9b680859a1a4c56afe33f5b1b3c0793
I20250629 01:51:44.913019 1155 tablet_bootstrap.cc:654] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:44.917732 1387 tablet.cc:1722] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:44.921571 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]) successfully deleted
I20250629 01:51:44.922049 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]). Expected version 2 got 0
W20250629 01:51:44.923211 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e83009c1599a43d18d1c73c0101ab0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e83009c1599a43d18d1c73c0101ab0be already in progress: creating tablet
W20250629 01:51:44.924365 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20586ba2af9e4cb7adec5ca6c3b76e42 already in progress: creating tablet
W20250629 01:51:44.925320 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4e354fce951943d69e4c666897f5caac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e354fce951943d69e4c666897f5caac already in progress: creating tablet
I20250629 01:51:44.925853 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 97439c6a6e844610b55b899966ae5b67 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:44.927929 1154 tablet_bootstrap.cc:654] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:44.930544 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1e22a0c02054c7db514eefe20ee3311 consensus state has no leader
W20250629 01:51:44.926707 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ef976703d61442f8987ff32070a33592 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef976703d61442f8987ff32070a33592 already in progress: creating tablet
I20250629 01:51:44.932448 1159 tablet_bootstrap.cc:492] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:44.932374 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0b16b6840c24484b90218fffc1da9621 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b16b6840c24484b90218fffc1da9621 already in progress: creating tablet
I20250629 01:51:44.933957 1159 tablet_bootstrap.cc:654] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:44.933369 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 68cedbd093b4420d8f7b29219ce8f77b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68cedbd093b4420d8f7b29219ce8f77b already in progress: creating tablet
W20250629 01:51:44.936833 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56b9ef027c9d42048e4a6dff221df793 consensus state has no leader
I20250629 01:51:44.936869 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_047550de222e41deaa74e50cd86bdea8" } new_table_name: "default.Table_3812c4a8d39844e69e8e142dbf156fae" modify_external_catalogs: true
I20250629 01:51:44.938022 1157 log.cc:1199] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/fca257d6cb614110948e1045dee44bdc
I20250629 01:51:44.938468 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d2344e007e2f4266aa706d32b9465347" } new_table_name: "default.Table_a9b6ef9e25bf4f8a9e3e8e0f9ce2060e" modify_external_catalogs: true
I20250629 01:51:44.939478 1382 tablet.cc:1722] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:44.940486 880 tablet_service.cc:1515] Processing DeleteTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.940902 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:44.946521 1157 ts_tablet_manager.cc:1939] T fca257d6cb614110948e1045dee44bdc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:44.947366 1576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:32789 every 8 connection(s)
I20250629 01:51:44.947696 1277 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:32789
I20250629 01:51:44.947883 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f3bf5fd9b674eeb8a5a029152cbae13 with delete_type TABLET_DATA_DELETED (replaced by tablet 942f7bbe29f94083b2494aab8fbcc0c5) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:44.948042 1455 raft_consensus.cc:491] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:44.948611 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 2 got 1
I20250629 01:51:44.949880 1277 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-2/data/info.pb
I20250629 01:51:44.951328 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:44.955389 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:44.960490 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d2c40cf88602426bbe61a1718e57da6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:44.961061 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 1277
W20250629 01:51:44.961127 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2c40cf88602426bbe61a1718e57da6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:44.963533 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 (DEFAULT_TABLE table=default.Table_98a1aa99dbcb4f448201bde47757f522 [id=5ee5e560e23a45ff897dbba0a0909cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.964413 1455 raft_consensus.cc:513] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:44.965029 1455 leader_election.cc:290] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:44.966575 745 tablet_service.cc:1468] Processing CreateTablet for tablet 8975c1deaaf44761be791f460b53ec74 (DEFAULT_TABLE table=default.Table_d532629a716e44368b4324cf8b41f873 [id=21d8cd304488494fb56540f29f71f25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.966728 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 (DEFAULT_TABLE table=default.Table_98a1aa99dbcb4f448201bde47757f522 [id=5ee5e560e23a45ff897dbba0a0909cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.967010 754 tablet_service.cc:1468] Processing CreateTablet for tablet f3bdc8bda8a940be865679a05b90bd08 (DEFAULT_TABLE table=default.Table_44f5d1913b0e4eedaac117acfdf02c22 [id=b6e2a7a8ccc245899af18bfd45e0f740]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.967108 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 798516e55ca747fcb2942026adb78af0 consensus state has no leader
I20250629 01:51:44.967465 756 tablet_service.cc:1468] Processing CreateTablet for tablet 9c6239dd09a343d59da51d114f48658a (DEFAULT_TABLE table=default.Table_e3283d56ec2743e2a1073ee8909e103f [id=c6251e55d21e4558a16574655b0be5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.968439 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 79e61d9576f8414ebf0074d57cdfb1f5 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:44.968775 710 leader_election.cc:343] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 5f3bf5fd9b674eeb8a5a029152cbae13
I20250629 01:51:44.970901 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.970934 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f3bdc8bda8a940be865679a05b90bd08 (DEFAULT_TABLE table=default.Table_44f5d1913b0e4eedaac117acfdf02c22 [id=b6e2a7a8ccc245899af18bfd45e0f740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.971138 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8a2d1a7f7146f4968590914374ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.971371 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3bdc8bda8a940be865679a05b90bd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.971431 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 8975c1deaaf44761be791f460b53ec74 (DEFAULT_TABLE table=default.Table_d532629a716e44368b4324cf8b41f873 [id=21d8cd304488494fb56540f29f71f25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.971719 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 9c6239dd09a343d59da51d114f48658a (DEFAULT_TABLE table=default.Table_e3283d56ec2743e2a1073ee8909e103f [id=c6251e55d21e4558a16574655b0be5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.972041 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8975c1deaaf44761be791f460b53ec74. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.973744 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6bf3c7d0bb548e0b77ba8b569a72d0d consensus state has no leader
I20250629 01:51:44.970989 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 (DEFAULT_TABLE table=default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.976222 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6239dd09a343d59da51d114f48658a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.976428 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe9674ffeb44ba6ab60de59d4c08670. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.976751 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b9b680859a1a4c56afe33f5b1b3c0793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9b680859a1a4c56afe33f5b1b3c0793 already in progress: creating tablet
I20250629 01:51:44.978137 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 97439c6a6e844610b55b899966ae5b67
I20250629 01:51:44.978500 451 catalog_manager.cc:6652] ReplaceTablet: tablet 97439c6a6e844610b55b899966ae5b67 deleted and replaced by tablet 1b0782c7e652419096710563930f2072
I20250629 01:51:44.980829 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 97439c6a6e844610b55b899966ae5b67 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b0782c7e652419096710563930f2072) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.981000 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3d87639255ec46d98fcfb0f9638ee39a (DEFAULT_TABLE table=default.Table_7acc3d53b2754aef87214be4a094b1f8 [id=4bbe205f87d54d69821df73deff787e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:44.981374 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 97439c6a6e844610b55b899966ae5b67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97439c6a6e844610b55b899966ae5b67 already in progress: creating tablet
I20250629 01:51:44.983300 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d87639255ec46d98fcfb0f9638ee39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.985342 1023 tablet_service.cc:1468] Processing CreateTablet for tablet cd9a41fae477470281dfba4d4e552b99 (DEFAULT_TABLE table=default.Table_f9284d7ccab1429d91910af2fd53b89c [id=4525666163d64ba981dfff1c095c6ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.989279 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8dd084685cc0407da9e12b0a4df386e9
I20250629 01:51:44.990160 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:44.990401 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd9a41fae477470281dfba4d4e552b99. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:44.990718 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8dd084685cc0407da9e12b0a4df386e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd084685cc0407da9e12b0a4df386e9 already in progress: creating tablet
I20250629 01:51:44.995460 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 45ca747f03524076b4112e4db78614fe (DEFAULT_TABLE table=default.Table_e39f46841d454041bb064e74d3c27bc0 [id=f7eba54966074342a3632852c544ff35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:44.995838 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45ca747f03524076b4112e4db78614fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:44.996515 1157 tablet_replica.cc:331] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.002354 885 tablet_service.cc:1468] Processing CreateTablet for tablet 9c6239dd09a343d59da51d114f48658a (DEFAULT_TABLE table=default.Table_e3283d56ec2743e2a1073ee8909e103f [id=c6251e55d21e4558a16574655b0be5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.004076 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.005661 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0d8a47c941e64eb1bd5332c0dbee8504"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.006119 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d8a47c941e64eb1bd5332c0dbee8504 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.007197 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af4bfaa5668e4c499de2890f2792c9eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.007632 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af4bfaa5668e4c499de2890f2792c9eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.008590 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_379f342e713042b49f4d21aa5e7b87a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.008680 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_93247e23895942578dd7e5f95a2cd9e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.009023 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_379f342e713042b49f4d21aa5e7b87a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.009207 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4c27cb96dae6400b892abad969dbeecb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.009611 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c27cb96dae6400b892abad969dbeecb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.010251 1157 raft_consensus.cc:2241] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:45.009025 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93247e23895942578dd7e5f95a2cd9e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.010277 895 tablet_service.cc:1468] Processing CreateTablet for tablet 8975c1deaaf44761be791f460b53ec74 (DEFAULT_TABLE table=default.Table_d532629a716e44368b4324cf8b41f873 [id=21d8cd304488494fb56540f29f71f25d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.010509 1157 raft_consensus.cc:2270] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.011286 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e39f46841d454041bb064e74d3c27bc0" } modify_external_catalogs: true
I20250629 01:51:45.011440 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e39f46841d454041bb064e74d3c27bc0" } modify_external_catalogs: true
I20250629 01:51:45.011996 1157 ts_tablet_manager.cc:1905] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.013341 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 (DEFAULT_TABLE table=default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.013944 886 tablet_service.cc:1468] Processing CreateTablet for tablet f3bdc8bda8a940be865679a05b90bd08 (DEFAULT_TABLE table=default.Table_44f5d1913b0e4eedaac117acfdf02c22 [id=b6e2a7a8ccc245899af18bfd45e0f740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.015918 880 tablet_service.cc:1468] Processing CreateTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 (DEFAULT_TABLE table=default.Table_98a1aa99dbcb4f448201bde47757f522 [id=5ee5e560e23a45ff897dbba0a0909cda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.014185 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f3bf5fd9b674eeb8a5a029152cbae13" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:45.017217 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 97439c6a6e844610b55b899966ae5b67 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b0782c7e652419096710563930f2072) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.017412 878 tablet_service.cc:1468] Processing CreateTablet for tablet cd9a41fae477470281dfba4d4e552b99 (DEFAULT_TABLE table=default.Table_f9284d7ccab1429d91910af2fd53b89c [id=4525666163d64ba981dfff1c095c6ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.017519 1154 tablet_bootstrap.cc:492] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.017711 1154 ts_tablet_manager.cc:1397] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.000s
I20250629 01:51:45.017982 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.018525 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3d87639255ec46d98fcfb0f9638ee39a (DEFAULT_TABLE table=default.Table_7acc3d53b2754aef87214be4a094b1f8 [id=4bbe205f87d54d69821df73deff787e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.018918 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6239dd09a343d59da51d114f48658a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.019505 1154 raft_consensus.cc:357] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.019655 1154 raft_consensus.cc:383] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.019707 1154 raft_consensus.cc:738] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.019894 1154 consensus_queue.cc:260] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.020323 1154 ts_tablet_manager.cc:1428] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:45.020605 1154 tablet_bootstrap.cc:492] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.023640 889 tablet_service.cc:1468] Processing CreateTablet for tablet 45ca747f03524076b4112e4db78614fe (DEFAULT_TABLE table=default.Table_e39f46841d454041bb064e74d3c27bc0 [id=f7eba54966074342a3632852c544ff35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.024020 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45ca747f03524076b4112e4db78614fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.025113 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_223fe61dd1034f42a871c3ff477bcf1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.025507 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_223fe61dd1034f42a871c3ff477bcf1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.026242 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8a2d1a7f7146f4968590914374ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.026496 447 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e] alter complete (version 2)
I20250629 01:51:45.028621 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 990d1fcdac1c406facb2da722421d7f4 (table default.Table_0685872de4674119b2386ac286878d35 [id=d02cba06b5bc499280a223f49faa9e40]). Expected version 2 got 1
I20250629 01:51:45.028890 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d87639255ec46d98fcfb0f9638ee39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.029340 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd9a41fae477470281dfba4d4e552b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.031137 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 05905638c6e241a095438f277f53adab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.031343 1158 tablet_replica.cc:331] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.031479 1158 raft_consensus.cc:2241] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.031622 1158 raft_consensus.cc:2270] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.032122 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe9674ffeb44ba6ab60de59d4c08670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.032248 1158 ts_tablet_manager.cc:1905] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.032805 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3bdc8bda8a940be865679a05b90bd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.036717 1154 tablet_bootstrap.cc:654] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.037550 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8975c1deaaf44761be791f460b53ec74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.037815 1158 ts_tablet_manager.cc:1918] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.037920 1158 log.cc:1199] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/05905638c6e241a095438f277f53adab
I20250629 01:51:45.037011 1339 tablet.cc:1722] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:45.038228 1158 ts_tablet_manager.cc:1939] T 05905638c6e241a095438f277f53adab P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.040303 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 05905638c6e241a095438f277f53adab (table default.Table_40038832d9f84a35b7db42595981b08d [id=151987e142c54e60b5137314e329dd82]) successfully deleted
I20250629 01:51:45.047113 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d1f5b20a8d5d4c7687aedcb1c9e54c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.047606 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1f5b20a8d5d4c7687aedcb1c9e54c2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.048694 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_661f7318ce384b5bad0d019f704d12c2" } new_table_name: "default.Table_cbbc85d10dc546b59c06dd6296eaa2db" modify_external_catalogs: true
I20250629 01:51:45.063128 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9914b6019e564cf7aa3ad9c511898fed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.063621 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9914b6019e564cf7aa3ad9c511898fed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.064183 1339 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:51:45.073436 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 52be184d0480495a894442d6f72a0f12 consensus state has no leader
I20250629 01:51:45.074683 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 45ca747f03524076b4112e4db78614fe
I20250629 01:51:45.081039 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.081355 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d6eb1cb53be64d2d81c86f31b21bbecc" } modify_external_catalogs: true
I20250629 01:51:45.081493 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d6eb1cb53be64d2d81c86f31b21bbecc" } modify_external_catalogs: true
W20250629 01:51:45.082044 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 45ca747f03524076b4112e4db78614fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45ca747f03524076b4112e4db78614fe already in progress: creating tablet
I20250629 01:51:45.083724 1157 ts_tablet_manager.cc:1918] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.084561 1154 tablet_bootstrap.cc:492] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.084583 1157 log.cc:1199] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5f3bf5fd9b674eeb8a5a029152cbae13
I20250629 01:51:45.084693 1154 ts_tablet_manager.cc:1397] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.012s
I20250629 01:51:45.085398 1154 raft_consensus.cc:357] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.085564 1154 raft_consensus.cc:383] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.085644 1154 raft_consensus.cc:738] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.085847 1154 consensus_queue.cc:260] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.086324 1154 ts_tablet_manager.cc:1428] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:45.087560 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8975c1deaaf44761be791f460b53ec74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.089820 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.090112 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3bdc8bda8a940be865679a05b90bd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.090314 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8a2d1a7f7146f4968590914374ba23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.090854 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e605f1f30c8d4fdd8503b71227359798" } modify_external_catalogs: true
I20250629 01:51:45.089183 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.091007 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e605f1f30c8d4fdd8503b71227359798" } modify_external_catalogs: true
I20250629 01:51:45.092327 1156 tablet_replica.cc:331] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.094741 1159 tablet_bootstrap.cc:492] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.094879 1159 ts_tablet_manager.cc:1397] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.162s user 0.017s sys 0.021s
I20250629 01:51:45.095796 1577 sys_catalog.cc:263] Verifying existing consensus state
I20250629 01:51:45.096114 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6239dd09a343d59da51d114f48658a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.096207 1548 raft_consensus.cc:491] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.096321 1548 raft_consensus.cc:513] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.097529 1548 leader_election.cc:290] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.101784 1157 ts_tablet_manager.cc:1939] T 5f3bf5fd9b674eeb8a5a029152cbae13 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.104876 1455 raft_consensus.cc:491] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.105013 1455 raft_consensus.cc:513] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:45.105158 748 tablet_service.cc:1515] Processing DeleteTablet for tablet d3a91740f16744439f55077c4fbadee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.105214 1577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Bootstrap starting.
I20250629 01:51:45.105347 755 tablet_service.cc:1468] Processing CreateTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 (DEFAULT_TABLE table=default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.105497 1455 leader_election.cc:290] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.105593 1097 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:45.105618 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 97439c6a6e844610b55b899966ae5b67 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b0782c7e652419096710563930f2072) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.105770 738 tablet_service.cc:1468] Processing CreateTablet for tablet 3d87639255ec46d98fcfb0f9638ee39a (DEFAULT_TABLE table=default.Table_7acc3d53b2754aef87214be4a094b1f8 [id=4bbe205f87d54d69821df73deff787e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.106060 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d87639255ec46d98fcfb0f9638ee39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.106094 740 tablet_service.cc:1515] Processing DeleteTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.106352 1159 raft_consensus.cc:357] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.106535 1159 raft_consensus.cc:383] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.106590 1159 raft_consensus.cc:738] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.106660 1156 raft_consensus.cc:2241] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.106827 1159 consensus_queue.cc:260] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.107077 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.107247 1159 ts_tablet_manager.cc:1428] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20250629 01:51:45.107463 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe9674ffeb44ba6ab60de59d4c08670. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.108345 747 tablet_service.cc:1468] Processing CreateTablet for tablet cd9a41fae477470281dfba4d4e552b99 (DEFAULT_TABLE table=default.Table_f9284d7ccab1429d91910af2fd53b89c [id=4525666163d64ba981dfff1c095c6ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.108630 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd9a41fae477470281dfba4d4e552b99. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.110345 850 leader_election.cc:343] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 05905638c6e241a095438f277f53adab
I20250629 01:51:45.110903 751 tablet_service.cc:1468] Processing CreateTablet for tablet 45ca747f03524076b4112e4db78614fe (DEFAULT_TABLE table=default.Table_e39f46841d454041bb064e74d3c27bc0 [id=f7eba54966074342a3632852c544ff35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.111258 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45ca747f03524076b4112e4db78614fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.112195 712 leader_election.cc:343] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.112423 712 leader_election.cc:304] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:45.112515 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.113231 1158 tablet_replica.cc:331] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:45.113247 850 consensus_peers.cc:489] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Couldn't send request to peer ef5c5979bb9e475f983af7c3e9e0e019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88031f8d8cbc44e4b8c53d9cd5360552. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:45.113483 1158 raft_consensus.cc:2241] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.113615 1158 raft_consensus.cc:2270] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.113948 1159 tablet_bootstrap.cc:492] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.114360 1548 raft_consensus.cc:491] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.114398 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_98a1aa99dbcb4f448201bde47757f522" } modify_external_catalogs: true
I20250629 01:51:45.114532 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_98a1aa99dbcb4f448201bde47757f522" } modify_external_catalogs: true
I20250629 01:51:45.114497 1548 raft_consensus.cc:513] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.115034 1548 leader_election.cc:290] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.115334 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1938e67acaf4ab19fcbed13e1f1bfc6" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:45.115428 1159 tablet_bootstrap.cc:654] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:45.115753 850 leader_election.cc:343] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.116588 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05905638c6e241a095438f277f53adab" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:45.117463 1156 raft_consensus.cc:2270] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:45.117676 710 leader_election.cc:343] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d3a91740f16744439f55077c4fbadee1
I20250629 01:51:45.117825 1455 raft_consensus.cc:2704] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
W20250629 01:51:45.118163 712 leader_election.cc:343] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: d3a91740f16744439f55077c4fbadee1
I20250629 01:51:45.118332 712 leader_election.cc:304] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:45.118476 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 346) took 1015 ms. Trace:
I20250629 01:51:45.119073 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1938e67acaf4ab19fcbed13e1f1bfc6" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:45.119508 1455 raft_consensus.cc:2747] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:45.118563 1157 rpcz_store.cc:276] 0629 01:51:44.103267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.103309 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:44.950846 (+847537us) ts_tablet_manager.cc:1145] Deleting tablet 5f3bf5fd9b674eeb8a5a029152cbae13
0629 01:51:44.952196 (+ 1350us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.078651 (+126455us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.083509 (+ 4858us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.118423 (+ 34914us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":847381}
I20250629 01:51:45.121043 1157 tablet_replica.cc:331] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:45.121085 850 leader_election.cc:343] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.121279 850 leader_election.cc:304] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:45.121510 1157 raft_consensus.cc:2241] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.121515 1548 raft_consensus.cc:2747] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:45.121624 850 leader_election.cc:343] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.121687 1157 raft_consensus.cc:2270] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.121723 850 leader_election.cc:304] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:45.121932 1548 raft_consensus.cc:2747] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:45.122130 1157 ts_tablet_manager.cc:1905] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.122946 1155 tablet_bootstrap.cc:492] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:45.123078 1155 ts_tablet_manager.cc:1397] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.212s user 0.013s sys 0.019s
I20250629 01:51:45.123730 1155 raft_consensus.cc:357] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.123885 1155 raft_consensus.cc:383] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.123942 1155 raft_consensus.cc:738] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.124150 1155 consensus_queue.cc:260] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.124691 1155 ts_tablet_manager.cc:1428] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:45.125003 1155 tablet_bootstrap.cc:492] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:45.126397 1155 tablet_bootstrap.cc:654] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.129333 1158 ts_tablet_manager.cc:1905] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.130851 1577 log.cc:826] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:45.134056 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_338563bee1704636a7021bf819690974" } new_table_name: "default.Table_c1fb2548e78248e98b54bfb7f01c0e51" modify_external_catalogs: true
I20250629 01:51:45.136031 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.136292 1158 ts_tablet_manager.cc:1918] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.136410 1158 log.cc:1199] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5e9f6de505a34afbae48a373b4b2a0cf
I20250629 01:51:45.136592 1156 ts_tablet_manager.cc:1905] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.136711 1158 ts_tablet_manager.cc:1939] T 5e9f6de505a34afbae48a373b4b2a0cf P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.140506 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5e9f6de505a34afbae48a373b4b2a0cf (table default.Table_bd59d9e2715745828993ae11d0dd56d0 [id=9e7864d3513b4aca8f2b2b2cf84805bf]) successfully deleted
I20250629 01:51:45.140924 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c61ad1c9857141faad549e9cc863671e" } new_table_name: "default.Table_7eb487cf3b984b829ea5456c141ea824" modify_external_catalogs: true
I20250629 01:51:45.142403 1156 ts_tablet_manager.cc:1918] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.142540 1156 log.cc:1199] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5f3bf5fd9b674eeb8a5a029152cbae13
I20250629 01:51:45.142634 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_98056b52f364497899c1998d23f58ce7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.143077 1156 ts_tablet_manager.cc:1939] T 5f3bf5fd9b674eeb8a5a029152cbae13 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:45.143083 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98056b52f364497899c1998d23f58ce7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.144035 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5f3bf5fd9b674eeb8a5a029152cbae13 (table default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]) successfully deleted
W20250629 01:51:45.145692 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 45ca747f03524076b4112e4db78614fe because the tablet was not found. No further retry: Not found: Tablet not found: 45ca747f03524076b4112e4db78614fe
I20250629 01:51:45.146204 1156 tablet_replica.cc:331] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.146584 1156 raft_consensus.cc:2241] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.146962 1156 raft_consensus.cc:2270] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.147603 1156 ts_tablet_manager.cc:1905] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.150076 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 05905638c6e241a095438f277f53adab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.150399 1154 tablet_bootstrap.cc:492] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.151716 1154 tablet_bootstrap.cc:654] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.152160 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 6f3a65c9c1c740b1865b96a6e2aea524 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.156041 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c712d978a3848b9bb3abdbce8314dec from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.163823 1156 ts_tablet_manager.cc:1918] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.164453 1159 tablet_bootstrap.cc:492] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.164646 1159 ts_tablet_manager.cc:1397] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.001s
I20250629 01:51:45.165472 1159 raft_consensus.cc:357] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.165720 1159 raft_consensus.cc:383] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.165848 1159 raft_consensus.cc:738] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.166110 1159 consensus_queue.cc:260] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.166790 1159 ts_tablet_manager.cc:1428] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:45.167042 1156 log.cc:1199] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d3a91740f16744439f55077c4fbadee1
I20250629 01:51:45.167409 1156 ts_tablet_manager.cc:1939] T d3a91740f16744439f55077c4fbadee1 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:45.168403 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca28c370cb5e435b9dc8db3efb233a44" } new_table_name: "default.Table_50eb1d5f0c5f41e0a975713cf15124ec" modify_external_catalogs: true
I20250629 01:51:45.170511 1157 ts_tablet_manager.cc:1918] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.170874 1159 tablet_bootstrap.cc:492] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.172049 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d3a91740f16744439f55077c4fbadee1 (table default.Table_e4165d1f8e364693a7dd96d2e5e47fa2 [id=cf0d80c2c9e2464db9ded94438df584a]) successfully deleted
I20250629 01:51:45.173524 1159 tablet_bootstrap.cc:654] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.173969 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7748846a3df14ffa801321ccde86e35f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:45.175558 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 97439c6a6e844610b55b899966ae5b67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97439c6a6e844610b55b899966ae5b67 already in progress: creating tablet
W20250629 01:51:45.176960 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b9b680859a1a4c56afe33f5b1b3c0793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9b680859a1a4c56afe33f5b1b3c0793 already in progress: creating tablet
I20250629 01:51:45.177038 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d34c7a4da774448f8a7e20900b74c956" } new_table_name: "default.Table_3f1b07c24c6441f4a4965f8e055648fe" modify_external_catalogs: true
I20250629 01:51:45.178619 1157 log.cc:1199] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/648fd1ca8f514ea3ba4554452b19933f
W20250629 01:51:45.178885 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8dd084685cc0407da9e12b0a4df386e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd084685cc0407da9e12b0a4df386e9 already in progress: creating tablet
I20250629 01:51:45.180543 1157 ts_tablet_manager.cc:1939] T 648fd1ca8f514ea3ba4554452b19933f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.181288 1156 tablet_replica.cc:331] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.181499 1156 raft_consensus.cc:2241] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.181690 1156 raft_consensus.cc:2270] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.182731 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 353) took 1062 ms. Trace:
I20250629 01:51:45.183373 1156 ts_tablet_manager.cc:1905] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.183297 1157 rpcz_store.cc:276] 0629 01:51:44.120333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.120379 (+ 46us) service_pool.cc:225] Handling call
0629 01:51:45.120964 (+1000585us) ts_tablet_manager.cc:1145] Deleting tablet 648fd1ca8f514ea3ba4554452b19933f
0629 01:51:45.120974 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.167457 (+ 46483us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.170389 (+ 2932us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.182702 (+ 12313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000417}
I20250629 01:51:45.183614 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 648fd1ca8f514ea3ba4554452b19933f (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]) successfully deleted
I20250629 01:51:45.183670 1157 tablet_replica.cc:331] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.183845 1157 raft_consensus.cc:2241] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.189716 1156 ts_tablet_manager.cc:1918] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.189822 1156 log.cc:1199] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/05905638c6e241a095438f277f53adab
I20250629 01:51:45.190455 1156 ts_tablet_manager.cc:1939] T 05905638c6e241a095438f277f53adab P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:45.191161 1155 tablet_bootstrap.cc:492] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:45.191282 1155 ts_tablet_manager.cc:1397] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.001s
I20250629 01:51:45.191267 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 05905638c6e241a095438f277f53adab (table default.Table_40038832d9f84a35b7db42595981b08d [id=151987e142c54e60b5137314e329dd82]) successfully deleted
I20250629 01:51:45.191983 1155 raft_consensus.cc:357] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.192154 1155 raft_consensus.cc:383] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.192235 1155 raft_consensus.cc:738] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.192420 1155 consensus_queue.cc:260] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.192852 1155 ts_tablet_manager.cc:1428] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:45.193091 1155 tablet_bootstrap.cc:492] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:45.194243 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:45.196689 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 5c6f3f19a4d441f58c8c6099e04f82ec from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.183979 1157 raft_consensus.cc:2270] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.193766 961 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:45.198014 1159 tablet_bootstrap.cc:492] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.198151 1159 ts_tablet_manager.cc:1397] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20250629 01:51:45.199476 1157 ts_tablet_manager.cc:1905] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.200809 1159 raft_consensus.cc:357] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.201017 1159 raft_consensus.cc:383] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.201087 1159 raft_consensus.cc:738] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.201361 1159 consensus_queue.cc:260] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.201841 1159 ts_tablet_manager.cc:1428] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:45.205380 1159 tablet_bootstrap.cc:492] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.207926 1155 tablet_bootstrap.cc:654] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.210583 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 (DEFAULT_TABLE table=default.Table_d2c40cf88602426bbe61a1718e57da6c [id=92d55fc2cf6343d7aac796eb0ddb93f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:45.210762 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2249d5cba1544653b088312e7d74802e consensus state has no leader
I20250629 01:51:45.210968 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f94d8fc1b6254ba69acadcaff3613d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.212478 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_de5d37756c7649fc9bf3ae6d5adf922f" } modify_external_catalogs: true
I20250629 01:51:45.212643 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_de5d37756c7649fc9bf3ae6d5adf922f" } modify_external_catalogs: true
W20250629 01:51:45.213644 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6d79ac3c5f04b5db295c93c3aae92cb consensus state has no leader
W20250629 01:51:45.213975 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf4b52fae47f45ad9d77b0a97a854860 consensus state has no leader
I20250629 01:51:45.216251 893 tablet_service.cc:1468] Processing CreateTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 (DEFAULT_TABLE table=default.Table_d2c40cf88602426bbe61a1718e57da6c [id=92d55fc2cf6343d7aac796eb0ddb93f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.216316 1027 tablet_service.cc:1468] Processing CreateTablet for tablet b15300fd8ca7489f9d77f1f8992f8df3 (DEFAULT_TABLE table=default.Table_12d6ac1cd45c43eeac32cc51d3aae591 [id=7e09582349d64d4fa089b53251ecf9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.216526 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.216594 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f94d8fc1b6254ba69acadcaff3613d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.216588 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b15300fd8ca7489f9d77f1f8992f8df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.220309 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d (DEFAULT_TABLE table=default.Table_6b2a627b815e45c480dcaafe356a5d50 [id=d5875760fd97485b88f3c69340dc7b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.220603 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.220705 895 tablet_service.cc:1468] Processing CreateTablet for tablet b15300fd8ca7489f9d77f1f8992f8df3 (DEFAULT_TABLE table=default.Table_12d6ac1cd45c43eeac32cc51d3aae591 [id=7e09582349d64d4fa089b53251ecf9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.220995 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b15300fd8ca7489f9d77f1f8992f8df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.231456 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_56ce186b15474e3ebf35ac8f454778bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.231931 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56ce186b15474e3ebf35ac8f454778bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.233624 1157 ts_tablet_manager.cc:1918] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.233754 1157 log.cc:1199] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f98bb48081984d23af55d7fa6a8bec7b
I20250629 01:51:45.234057 1157 ts_tablet_manager.cc:1939] T f98bb48081984d23af55d7fa6a8bec7b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.234201 455 master_service.cc:948] ReplaceTablet: received request to replace tablet cabb0298771e4fd5be29e530d3a8ed3d from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.234336 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 355) took 1113 ms. Trace:
I20250629 01:51:45.234408 1157 rpcz_store.cc:276] 0629 01:51:44.121034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.121453 (+ 419us) service_pool.cc:225] Handling call
0629 01:51:45.183621 (+1062168us) ts_tablet_manager.cc:1145] Deleting tablet f98bb48081984d23af55d7fa6a8bec7b
0629 01:51:45.183630 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.210425 (+ 26795us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.233546 (+ 23121us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.234296 (+ 750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061521}
I20250629 01:51:45.234676 1157 tablet_replica.cc:331] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.234845 1157 raft_consensus.cc:2241] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.234814 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0782c7e652419096710563930f2072 (DEFAULT_TABLE table=default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.234997 1157 raft_consensus.cc:2270] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.235147 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0782c7e652419096710563930f2072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.235425 1157 ts_tablet_manager.cc:1905] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.236155 878 tablet_service.cc:1468] Processing CreateTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d (DEFAULT_TABLE table=default.Table_6b2a627b815e45c480dcaafe356a5d50 [id=d5875760fd97485b88f3c69340dc7b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.236528 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.239665 886 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0782c7e652419096710563930f2072 (DEFAULT_TABLE table=default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.240370 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0782c7e652419096710563930f2072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.244998 745 tablet_service.cc:1468] Processing CreateTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 (DEFAULT_TABLE table=default.Table_d2c40cf88602426bbe61a1718e57da6c [id=92d55fc2cf6343d7aac796eb0ddb93f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.246882 1577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=45 ignored=0} inserts{seen=46 ignored=0} mutations{seen=31 ignored=0} orphaned_commits=0. Pending: 45 replicates
I20250629 01:51:45.247212 1154 tablet_bootstrap.cc:492] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.247342 1154 ts_tablet_manager.cc:1397] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.005s
I20250629 01:51:45.247603 1577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: Bootstrap complete.
I20250629 01:51:45.247979 1154 raft_consensus.cc:357] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.248160 1154 raft_consensus.cc:383] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.248230 1154 raft_consensus.cc:738] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.248466 1154 consensus_queue.cc:260] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.251140 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f98bb48081984d23af55d7fa6a8bec7b (table default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]) successfully deleted
W20250629 01:51:45.252434 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daf312e60c014fe698d3ec9f1a1b769c consensus state has no leader
I20250629 01:51:45.252753 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.252689 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_06317866df174e2db51b073ad43db5db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.253166 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06317866df174e2db51b073ad43db5db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.253347 738 tablet_service.cc:1468] Processing CreateTablet for tablet 1b0782c7e652419096710563930f2072 (DEFAULT_TABLE table=default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.252738 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ecd89721aae04a9abed2f4ff44cfd3dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.254680 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecd89721aae04a9abed2f4ff44cfd3dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.255309 1577 raft_consensus.cc:357] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Replica starting. Triggering 45 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
W20250629 01:51:45.256677 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f98bb48081984d23af55d7fa6a8bec7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f98bb48081984d23af55d7fa6a8bec7b already in progress: creating tablet
I20250629 01:51:45.256904 751 tablet_service.cc:1468] Processing CreateTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d (DEFAULT_TABLE table=default.Table_6b2a627b815e45c480dcaafe356a5d50 [id=d5875760fd97485b88f3c69340dc7b01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.259001 753 tablet_service.cc:1468] Processing CreateTablet for tablet b15300fd8ca7489f9d77f1f8992f8df3 (DEFAULT_TABLE table=default.Table_12d6ac1cd45c43eeac32cc51d3aae591 [id=7e09582349d64d4fa089b53251ecf9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.259744 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 648fd1ca8f514ea3ba4554452b19933f with delete_type TABLET_DATA_DELETED (replaced by tablet 0b7a640451934bff8672cd88dc86c529) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.260437 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f94d8fc1b6254ba69acadcaff3613d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.260732 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.260776 1156 tablet_replica.cc:331] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.261178 1156 raft_consensus.cc:2241] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.261330 1156 raft_consensus.cc:2270] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.261500 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b15300fd8ca7489f9d77f1f8992f8df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.264256 1156 ts_tablet_manager.cc:1905] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.268239 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b0782c7e652419096710563930f2072. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.269290 1157 ts_tablet_manager.cc:1918] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.276178 1157 log.cc:1199] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5e9f6de505a34afbae48a373b4b2a0cf
I20250629 01:51:45.276577 1157 ts_tablet_manager.cc:1939] T 5e9f6de505a34afbae48a373b4b2a0cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.276942 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 354) took 1156 ms. Trace:
I20250629 01:51:45.277101 1157 rpcz_store.cc:276] 0629 01:51:44.120424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.120574 (+ 150us) service_pool.cc:225] Handling call
0629 01:51:45.234617 (+1114043us) ts_tablet_manager.cc:1145] Deleting tablet 5e9f6de505a34afbae48a373b4b2a0cf
0629 01:51:45.234627 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.238251 (+ 3624us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.269190 (+ 30939us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.276913 (+ 7723us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112413}
I20250629 01:51:45.275086 1159 tablet_bootstrap.cc:654] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.277612 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 5e9f6de505a34afbae48a373b4b2a0cf (table default.Table_bd59d9e2715745828993ae11d0dd56d0 [id=9e7864d3513b4aca8f2b2b2cf84805bf]) successfully deleted
I20250629 01:51:45.282348 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 358) took 1153 ms. Trace:
I20250629 01:51:45.282461 1157 rpcz_store.cc:276] 0629 01:51:44.128781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.128929 (+ 148us) service_pool.cc:225] Handling call
0629 01:51:45.282235 (+1153306us) ts_tablet_manager.cc:1145] Deleting tablet 0b7a640451934bff8672cd88dc86c529
0629 01:51:45.282242 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.282318 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1152934}
W20250629 01:51:45.282824 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0b7a640451934bff8672cd88dc86c529 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b7a640451934bff8672cd88dc86c529 already in progress: creating tablet
I20250629 01:51:45.283138 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 360) took 1147 ms. Trace:
I20250629 01:51:45.284750 1156 ts_tablet_manager.cc:1918] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.284870 1156 log.cc:1199] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/648fd1ca8f514ea3ba4554452b19933f
I20250629 01:51:45.284890 1157 rpcz_store.cc:276] 0629 01:51:44.135955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.136013 (+ 58us) service_pool.cc:225] Handling call
0629 01:51:45.283024 (+1147011us) ts_tablet_manager.cc:1145] Deleting tablet 3db333c60cb3485a9daae316b9f1a8c8
0629 01:51:45.283032 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.283109 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146255}
I20250629 01:51:45.285179 1156 ts_tablet_manager.cc:1939] T 648fd1ca8f514ea3ba4554452b19933f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:45.285444 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 378) took 1019 ms. Trace:
I20250629 01:51:45.285673 1157 rpcz_store.cc:276] 0629 01:51:44.266219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.284661 (+ 18442us) service_pool.cc:225] Handling call
0629 01:51:45.285342 (+1000681us) ts_tablet_manager.cc:1145] Deleting tablet a6a012ba891949538800581436ef5380
0629 01:51:45.285351 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.285421 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":915292}
I20250629 01:51:45.286088 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 648fd1ca8f514ea3ba4554452b19933f (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]) successfully deleted
I20250629 01:51:45.286336 1157 tablet_replica.cc:331] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.286729 1157 raft_consensus.cc:2241] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:45.286953 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1b0782c7e652419096710563930f2072
I20250629 01:51:45.287279 1157 raft_consensus.cc:2270] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.287597 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_be093346e01e4a688e5ede68f9e96fbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.287770 1157 ts_tablet_manager.cc:1905] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:45.288082 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be093346e01e4a688e5ede68f9e96fbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.289229 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:45.289638 455 catalog_manager.cc:6652] ReplaceTablet: tablet cabb0298771e4fd5be29e530d3a8ed3d deleted and replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd
I20250629 01:51:45.289966 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_768dd2c93bfa4a828037ed23b11dc5ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.290370 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_768dd2c93bfa4a828037ed23b11dc5ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.290464 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c0b9afb9976541b2bc7ab188f35a62ec
I20250629 01:51:45.291074 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_74ed2fc8d5de4c46bbce4736df36a2be" } new_table_name: "default.Table_c7d1c2fe02d04423803f16f69f0d6625" modify_external_catalogs: true
I20250629 01:51:45.292142 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5b8a2d1a7f7146f4968590914374ba23
I20250629 01:51:45.292726 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a9fddce63afb42b4be73a8c4807e050a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.293118 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9fddce63afb42b4be73a8c4807e050a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.293767 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b0782c7e652419096710563930f2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.293985 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.294188 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.294399 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6f3a65c9c1c740b1865b96a6e2aea524
W20250629 01:51:45.294580 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1b0782c7e652419096710563930f2072 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b0782c7e652419096710563930f2072 already in progress: creating tablet
W20250629 01:51:45.294793 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5b8a2d1a7f7146f4968590914374ba23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b8a2d1a7f7146f4968590914374ba23 already in progress: creating tablet
I20250629 01:51:45.294795 454 catalog_manager.cc:6652] ReplaceTablet: tablet 6f3a65c9c1c740b1865b96a6e2aea524 deleted and replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b
I20250629 01:51:45.295316 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.295655 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_640044e112e346b4bbbd17ba7dee16a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.296046 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_640044e112e346b4bbbd17ba7dee16a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.296871 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.297688 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:45.297848 745 tablet_service.cc:1515] Processing DeleteTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.298045 738 tablet_service.cc:1515] Processing DeleteTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d with delete_type TABLET_DATA_DELETED (replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.298226 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b0782c7e652419096710563930f2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.298250 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_821e341e379740a59615270511038004" } modify_external_catalogs: true
I20250629 01:51:45.298378 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_821e341e379740a59615270511038004" } modify_external_catalogs: true
I20250629 01:51:45.299547 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5c6f3f19a4d441f58c8c6099e04f82ec
I20250629 01:51:45.299762 1157 ts_tablet_manager.cc:1918] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250629 01:51:45.299863 1157 log.cc:1199] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/88031f8d8cbc44e4b8c53d9cd5360552
I20250629 01:51:45.299889 446 catalog_manager.cc:6652] ReplaceTablet: tablet 5c6f3f19a4d441f58c8c6099e04f82ec deleted and replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155
I20250629 01:51:45.300194 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fceee991e4084064b9ec489915a42d1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.300577 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fceee991e4084064b9ec489915a42d1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.300777 1159 tablet_bootstrap.cc:492] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.300876 1159 ts_tablet_manager.cc:1397] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.004s
I20250629 01:51:45.301554 1159 raft_consensus.cc:357] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.301756 1159 raft_consensus.cc:383] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.301863 1159 raft_consensus.cc:738] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.302030 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c712d978a3848b9bb3abdbce8314dec from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.302062 1159 consensus_queue.cc:260] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.302493 1159 ts_tablet_manager.cc:1428] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250629 01:51:45.295668 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5b8a2d1a7f7146f4968590914374ba23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b8a2d1a7f7146f4968590914374ba23 already in progress: creating tablet
I20250629 01:51:45.297888 1577 raft_consensus.cc:738] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f304077528b34c61857a659e5873631d, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.303447 1159 tablet_bootstrap.cc:492] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:45.303210 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6f3a65c9c1c740b1865b96a6e2aea524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f3a65c9c1c740b1865b96a6e2aea524 already in progress: creating tablet
I20250629 01:51:45.303905 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec with delete_type TABLET_DATA_DELETED (replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:45.304026 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6f3a65c9c1c740b1865b96a6e2aea524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f3a65c9c1c740b1865b96a6e2aea524 already in progress: creating tablet
W20250629 01:51:45.304204 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c0b9afb9976541b2bc7ab188f35a62ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0b9afb9976541b2bc7ab188f35a62ec already in progress: creating tablet
W20250629 01:51:45.304378 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet cabb0298771e4fd5be29e530d3a8ed3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cabb0298771e4fd5be29e530d3a8ed3d already in progress: creating tablet
W20250629 01:51:45.304509 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1b0782c7e652419096710563930f2072 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b0782c7e652419096710563930f2072 already in progress: creating tablet
I20250629 01:51:45.304967 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.305393 1577 consensus_queue.cc:260] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:45.305563 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec with delete_type TABLET_DATA_DELETED (replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.305773 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:45.306553 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3db333c60cb3485a9daae316b9f1a8c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3db333c60cb3485a9daae316b9f1a8c8 already in progress: creating tablet
W20250629 01:51:45.306774 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a6a012ba891949538800581436ef5380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6a012ba891949538800581436ef5380 already in progress: creating tablet
I20250629 01:51:45.307129 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b0782c7e652419096710563930f2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.307320 895 tablet_service.cc:1515] Processing DeleteTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d with delete_type TABLET_DATA_DELETED (replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.307503 878 tablet_service.cc:1515] Processing DeleteTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.307651 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.307833 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.308029 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.308236 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec with delete_type TABLET_DATA_DELETED (replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:45.309736 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5c6f3f19a4d441f58c8c6099e04f82ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c6f3f19a4d441f58c8c6099e04f82ec already in progress: creating tablet
W20250629 01:51:45.309958 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4a692f00ba244a80b9a6d07029f8d27e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a692f00ba244a80b9a6d07029f8d27e already in progress: creating tablet
W20250629 01:51:45.310164 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5c6f3f19a4d441f58c8c6099e04f82ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c6f3f19a4d441f58c8c6099e04f82ec already in progress: creating tablet
W20250629 01:51:45.310372 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4a692f00ba244a80b9a6d07029f8d27e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a692f00ba244a80b9a6d07029f8d27e already in progress: creating tablet
I20250629 01:51:45.311611 883 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.316761 1159 tablet_bootstrap.cc:654] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.319227 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d with delete_type TABLET_DATA_DELETED (replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.319772 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:45.319792 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet cabb0298771e4fd5be29e530d3a8ed3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cabb0298771e4fd5be29e530d3a8ed3d already in progress: creating tablet
W20250629 01:51:45.320307 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c0b9afb9976541b2bc7ab188f35a62ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0b9afb9976541b2bc7ab188f35a62ec already in progress: creating tablet
I20250629 01:51:45.323089 1577 sys_catalog.cc:564] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: configured and running, proceeding with master startup.
I20250629 01:51:45.324914 1598 sys_catalog.cc:455] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:45.325238 1598 sys_catalog.cc:458] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:45.325654 1157 ts_tablet_manager.cc:1939] T 88031f8d8cbc44e4b8c53d9cd5360552 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:45.335017 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d4320f615464fedbcab25137952f684 consensus state has no leader
I20250629 01:51:45.337301 1157 tablet_replica.cc:331] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.337319 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 88031f8d8cbc44e4b8c53d9cd5360552 (table default.Table_328919206d1e44838bc19f8a2ff9d371 [id=ac80888442774af495a53bba44dfa763]) successfully deleted
I20250629 01:51:45.337524 1157 raft_consensus.cc:2241] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
W20250629 01:51:45.337669 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8bcc3f2a76bc4bae8b306b54e75656cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bcc3f2a76bc4bae8b306b54e75656cc already in progress: creating tablet
I20250629 01:51:45.337822 1157 pending_rounds.cc:70] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Trying to abort 2 pending ops.
I20250629 01:51:45.337920 1157 pending_rounds.cc:77] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7172759163034054656 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "9c7f1f2485cb4bf3b8e0dbdd60272b7f" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_386bd2d6e963443cb19c211c4e6e0120" dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" new_extra_config { } }
W20250629 01:51:45.338038 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 75ce1bd4af434f7e99092aafc0ab0118 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75ce1bd4af434f7e99092aafc0ab0118 already in progress: creating tablet
W20250629 01:51:45.338212 1157 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:56376: Aborted: Op aborted
W20250629 01:51:45.338322 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f already in progress: creating tablet
I20250629 01:51:45.338538 1157 pending_rounds.cc:77] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759157226446848 op_type: NO_OP noop_request { }
I20250629 01:51:45.338654 1157 raft_consensus.cc:2887] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:45.338749 1157 raft_consensus.cc:2270] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.339357 1159 tablet_bootstrap.cc:492] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.339480 1159 ts_tablet_manager.cc:1397] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20250629 01:51:45.340023 1159 raft_consensus.cc:357] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:45.340238 1159 raft_consensus.cc:383] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.340322 1159 raft_consensus.cc:738] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.340538 1159 consensus_queue.cc:260] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:45.340994 1159 ts_tablet_manager.cc:1428] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:45.342514 1155 tablet_bootstrap.cc:492] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:45.342522 1159 tablet_bootstrap.cc:492] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.342679 1155 ts_tablet_manager.cc:1397] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.150s user 0.007s sys 0.000s
I20250629 01:51:45.343364 1155 raft_consensus.cc:357] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.343555 1155 raft_consensus.cc:383] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.343644 1155 raft_consensus.cc:738] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.344590 1159 tablet_bootstrap.cc:654] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.347342 1155 consensus_queue.cc:260] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.347126 1157 ts_tablet_manager.cc:1905] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.352206 1155 ts_tablet_manager.cc:1428] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20250629 01:51:45.352838 961 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:45.356420 1155 tablet_bootstrap.cc:492] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:45.358044 1155 tablet_bootstrap.cc:654] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.359090 1154 ts_tablet_manager.cc:1428] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.112s user 0.002s sys 0.000s
I20250629 01:51:45.360491 753 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.360740 1156 tablet_replica.cc:331] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.360885 1156 raft_consensus.cc:2241] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.361047 1156 raft_consensus.cc:2270] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.361783 1156 ts_tablet_manager.cc:1905] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.368539 1156 ts_tablet_manager.cc:1918] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.368696 1156 log.cc:1199] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f98bb48081984d23af55d7fa6a8bec7b
I20250629 01:51:45.369102 1156 ts_tablet_manager.cc:1939] T f98bb48081984d23af55d7fa6a8bec7b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:45.371470 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f98bb48081984d23af55d7fa6a8bec7b (table default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]) successfully deleted
I20250629 01:51:45.373026 753 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:45.373766 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f98bb48081984d23af55d7fa6a8bec7b because the tablet was not found. No further retry: Not found: Tablet not found: f98bb48081984d23af55d7fa6a8bec7b
I20250629 01:51:45.374228 1154 tablet_bootstrap.cc:492] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.375648 1154 tablet_bootstrap.cc:654] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.379528 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f98bb48081984d23af55d7fa6a8bec7b with delete_type TABLET_DATA_DELETED (replaced by tablet 5f60c48c924d49a186a8c81d154bdc54) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.379791 1158 tablet_replica.cc:331] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.379982 1158 raft_consensus.cc:2241] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.380182 1158 raft_consensus.cc:2270] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.380836 1158 ts_tablet_manager.cc:1905] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.382242 1159 tablet_bootstrap.cc:492] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.382391 1159 ts_tablet_manager.cc:1397] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.001s
I20250629 01:51:45.383096 1159 raft_consensus.cc:357] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.383224 1159 raft_consensus.cc:383] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.383283 1159 raft_consensus.cc:738] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.383450 1159 consensus_queue.cc:260] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.383723 1154 tablet_bootstrap.cc:492] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.383831 1048 raft_consensus.cc:3058] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:45.383847 1154 ts_tablet_manager.cc:1397] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20250629 01:51:45.384593 1154 raft_consensus.cc:357] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.384747 1154 raft_consensus.cc:383] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.384836 1154 raft_consensus.cc:738] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.385875 1159 ts_tablet_manager.cc:1428] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:45.386194 1159 tablet_bootstrap.cc:492] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.385015 1154 consensus_queue.cc:260] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.387043 1455 consensus_queue.cc:1035] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:45.387519 1548 raft_consensus.cc:491] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.387593 1159 tablet_bootstrap.cc:654] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.387640 1548 raft_consensus.cc:513] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.388188 1548 leader_election.cc:290] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.389907 1154 ts_tablet_manager.cc:1428] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:51:45.390425 1157 ts_tablet_manager.cc:1918] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:45.390573 1157 log.cc:1199] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9c7f1f2485cb4bf3b8e0dbdd60272b7f
I20250629 01:51:45.390930 1157 ts_tablet_manager.cc:1939] T 9c7f1f2485cb4bf3b8e0dbdd60272b7f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.391196 1097 heartbeater.cc:344] Connected to a master server at 127.0.104.60:32789
I20250629 01:51:45.392362 1157 tablet_replica.cc:331] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.392553 1157 raft_consensus.cc:2241] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.392885 455 master_service.cc:948] ReplaceTablet: received request to replace tablet ef976703d61442f8987ff32070a33592 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.392905 1157 raft_consensus.cc:2270] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.393321 1157 ts_tablet_manager.cc:1905] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.394225 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829c64e2c71347f4af2c8870cdd8af31" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:45.394456 1154 tablet_bootstrap.cc:492] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.395153 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "829c64e2c71347f4af2c8870cdd8af31" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:45.395820 1154 tablet_bootstrap.cc:654] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.403311 1154 tablet_bootstrap.cc:492] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.403782 1154 ts_tablet_manager.cc:1397] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250629 01:51:45.405364 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7f95c64f736b4aee86abc0c7785ab8d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.406132 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f95c64f736b4aee86abc0c7785ab8d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.407925 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9c7f1f2485cb4bf3b8e0dbdd60272b7f (table default.Table_386bd2d6e963443cb19c211c4e6e0120 [id=08450a0d665043f8a1513dbb72d9edea]) successfully deleted
W20250629 01:51:45.408401 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 648b0a0f742b4e7da9eaff6bd8795fc4 already in progress: creating tablet
W20250629 01:51:45.400840 850 leader_election.cc:343] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:45.408675 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 942f7bbe29f94083b2494aab8fbcc0c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 942f7bbe29f94083b2494aab8fbcc0c5 already in progress: creating tablet
W20250629 01:51:45.408802 850 leader_election.cc:343] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:45.408877 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet dfee69f9484046d8abe10d2f07033e09 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfee69f9484046d8abe10d2f07033e09 already in progress: creating tablet
I20250629 01:51:45.408895 850 leader_election.cc:304] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:45.409111 1548 raft_consensus.cc:2747] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:45.408630 1159 tablet_bootstrap.cc:492] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.409521 1159 ts_tablet_manager.cc:1397] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250629 01:51:45.410136 1159 raft_consensus.cc:357] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.410332 1159 raft_consensus.cc:383] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.410426 1159 raft_consensus.cc:738] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.410674 1159 consensus_queue.cc:260] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.411077 1159 ts_tablet_manager.cc:1428] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:45.411345 1159 tablet_bootstrap.cc:492] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:45.412181 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4e354fce951943d69e4c666897f5caac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e354fce951943d69e4c666897f5caac already in progress: creating tablet
W20250629 01:51:45.412483 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20586ba2af9e4cb7adec5ca6c3b76e42 already in progress: creating tablet
I20250629 01:51:45.412791 1159 tablet_bootstrap.cc:654] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:45.412767 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e83009c1599a43d18d1c73c0101ab0be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e83009c1599a43d18d1c73c0101ab0be already in progress: creating tablet
I20250629 01:51:45.417044 1154 raft_consensus.cc:357] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.419727 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:45.424400 1465 consensus_queue.cc:1035] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:45.429581 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 575006c12f7c41658e234f7c6c425d69 with delete_type TABLET_DATA_DELETED (replaced by tablet a792a26725954adea15ad57e31bfb224) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.429786 1158 ts_tablet_manager.cc:1918] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.430014 1158 log.cc:1199] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f98bb48081984d23af55d7fa6a8bec7b
I20250629 01:51:45.430246 1154 raft_consensus.cc:383] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.430337 1154 raft_consensus.cc:738] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.430596 1154 consensus_queue.cc:260] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.431051 1154 ts_tablet_manager.cc:1428] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250629 01:51:45.431339 1154 tablet_bootstrap.cc:492] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.432730 1154 tablet_bootstrap.cc:654] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.434381 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_abdf20a022384ea199a07cc08b8b9f97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.434849 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abdf20a022384ea199a07cc08b8b9f97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.436998 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7a3f650c91894a0090838ca98207ee9d" } new_table_name: "default.Table_bf7b726d4c38474994e10c22d1aa6076" modify_external_catalogs: true
I20250629 01:51:45.438980 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_58dcfb06bfe149d28ad21cc67702f6d5" } new_table_name: "default.Table_4c71f3d9eabc42539dbe4ed7cbf6d738" modify_external_catalogs: true
I20250629 01:51:45.441416 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9b58ddddb14343dcb197eff4c517689c" } new_table_name: "default.Table_a8bede9e061941eaabacba34109b16ab" modify_external_catalogs: true
I20250629 01:51:45.442193 1159 tablet_bootstrap.cc:492] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.442353 1159 ts_tablet_manager.cc:1397] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250629 01:51:45.442430 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 648fd1ca8f514ea3ba4554452b19933f with delete_type TABLET_DATA_DELETED (replaced by tablet 0b7a640451934bff8672cd88dc86c529) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:45.442726 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ef976703d61442f8987ff32070a33592 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef976703d61442f8987ff32070a33592 already in progress: creating tablet
I20250629 01:51:45.443311 1158 ts_tablet_manager.cc:1939] T f98bb48081984d23af55d7fa6a8bec7b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.443287 1159 raft_consensus.cc:357] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.443543 1159 raft_consensus.cc:383] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.443739 1159 raft_consensus.cc:738] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.444059 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f98bb48081984d23af55d7fa6a8bec7b (table default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]) successfully deleted
I20250629 01:51:45.444343 1159 consensus_queue.cc:260] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.446583 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:45.446812 1159 ts_tablet_manager.cc:1428] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:45.447041 1159 tablet_bootstrap.cc:492] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.447973 1158 tablet_replica.cc:331] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.448480 1159 tablet_bootstrap.cc:654] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.448621 1622 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250629 01:51:45.452903 1157 ts_tablet_manager.cc:1918] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.456646 1157 log.cc:1199] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/05905638c6e241a095438f277f53adab
I20250629 01:51:45.457019 1157 ts_tablet_manager.cc:1939] T 05905638c6e241a095438f277f53adab P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.457461 1158 raft_consensus.cc:2241] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.457597 1158 raft_consensus.cc:2270] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.458412 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4c27cb96dae6400b892abad969dbeecb" } new_table_name: "default.Table_209b7bc3f1de4919a043f2ab3c1c5d6a" modify_external_catalogs: true
I20250629 01:51:45.458371 883 tablet_service.cc:1468] Processing CreateTablet for tablet a014907d09c94745859436f1eb73b280 (DEFAULT_TABLE table=default.Table_98056b52f364497899c1998d23f58ce7 [id=d63189e5f0564d20a7f992ec881309fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.459829 1013 tablet_service.cc:1468] Processing CreateTablet for tablet a014907d09c94745859436f1eb73b280 (DEFAULT_TABLE table=default.Table_98056b52f364497899c1998d23f58ce7 [id=d63189e5f0564d20a7f992ec881309fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.460184 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a014907d09c94745859436f1eb73b280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.461822 1158 ts_tablet_manager.cc:1905] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.462467 893 tablet_service.cc:1468] Processing CreateTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 (DEFAULT_TABLE table=default.Table_af4bfaa5668e4c499de2890f2792c9eb [id=64e6f3a1044a485dac278b2deccc979d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.462800 880 tablet_service.cc:1468] Processing CreateTablet for tablet ce90c045115844b0b96143f91f1be1d4 (DEFAULT_TABLE table=default.Table_379f342e713042b49f4d21aa5e7b87a7 [id=7cb4f3fa847544c0852c62f40a1c9ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.463166 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8295e8178eff4d5eb95ac885092731e0 (DEFAULT_TABLE table=default.Table_0d8a47c941e64eb1bd5332c0dbee8504 [id=c17c3600bb2447d69097c8223ce973f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.464546 878 tablet_service.cc:1468] Processing CreateTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a (DEFAULT_TABLE table=default.Table_4c27cb96dae6400b892abad969dbeecb [id=d62355a430724b96abb294c451c9bd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.464883 895 tablet_service.cc:1468] Processing CreateTablet for tablet a12fffe61d7140308acd181bfa3d2700 (DEFAULT_TABLE table=default.Table_223fe61dd1034f42a871c3ff477bcf1e [id=075cbade2bd84a68a76f354b7796c991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.465554 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f (DEFAULT_TABLE table=default.Table_93247e23895942578dd7e5f95a2cd9e4 [id=2ba8a990fc3a474aac09caa7380ccb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.466341 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9d2aa33a5c4d9ebecaa83c32b39dcb (DEFAULT_TABLE table=default.Table_d1f5b20a8d5d4c7687aedcb1c9e54c2c [id=7a043339924b4d5a814f8aa156943aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.467906 753 tablet_service.cc:1468] Processing CreateTablet for tablet a014907d09c94745859436f1eb73b280 (DEFAULT_TABLE table=default.Table_98056b52f364497899c1998d23f58ce7 [id=d63189e5f0564d20a7f992ec881309fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.468277 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a014907d09c94745859436f1eb73b280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.471719 1158 ts_tablet_manager.cc:1918] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.471825 1158 log.cc:1199] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/575006c12f7c41658e234f7c6c425d69
I20250629 01:51:45.472594 885 tablet_service.cc:1468] Processing CreateTablet for tablet 303f77a729c648f9b1146d11056b242d (DEFAULT_TABLE table=default.Table_9914b6019e564cf7aa3ad9c511898fed [id=7c4d6f0956f142d195bd44428072be8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.473424 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d6f1eb43b24fda94aa0afcf4501195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.474709 1013 tablet_service.cc:1468] Processing CreateTablet for tablet ce90c045115844b0b96143f91f1be1d4 (DEFAULT_TABLE table=default.Table_379f342e713042b49f4d21aa5e7b87a7 [id=7cb4f3fa847544c0852c62f40a1c9ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.475013 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce90c045115844b0b96143f91f1be1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.477716 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 327c303b6b9347f5b4cb78f74d41a44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.477650 738 tablet_service.cc:1468] Processing CreateTablet for tablet ce90c045115844b0b96143f91f1be1d4 (DEFAULT_TABLE table=default.Table_379f342e713042b49f4d21aa5e7b87a7 [id=7cb4f3fa847544c0852c62f40a1c9ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.477998 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce90c045115844b0b96143f91f1be1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.478013 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303f77a729c648f9b1146d11056b242d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.478525 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.478899 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8295e8178eff4d5eb95ac885092731e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.479272 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a014907d09c94745859436f1eb73b280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.483544 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8295e8178eff4d5eb95ac885092731e0 (DEFAULT_TABLE table=default.Table_0d8a47c941e64eb1bd5332c0dbee8504 [id=c17c3600bb2447d69097c8223ce973f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.483532 1466 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" instance_seqno: 1751161898172646) as {username='slave'} at 127.0.104.2:49167; Asking this server to re-register.
I20250629 01:51:45.483907 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8295e8178eff4d5eb95ac885092731e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.484601 961 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:45.487285 1466 ts_manager.cc:194] Registered new tserver with Master: 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:45.488937 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 (DEFAULT_TABLE table=default.Table_af4bfaa5668e4c499de2890f2792c9eb [id=64e6f3a1044a485dac278b2deccc979d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.489315 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d6f1eb43b24fda94aa0afcf4501195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.491382 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9d2aa33a5c4d9ebecaa83c32b39dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.491735 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12fffe61d7140308acd181bfa3d2700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.492151 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce90c045115844b0b96143f91f1be1d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.493285 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f6acf9f470204fdb876176dc5c191895" } modify_external_catalogs: true
I20250629 01:51:45.493430 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f6acf9f470204fdb876176dc5c191895" } modify_external_catalogs: true
I20250629 01:51:45.495849 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_640044e112e346b4bbbd17ba7dee16a1" } modify_external_catalogs: true
I20250629 01:51:45.496006 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_640044e112e346b4bbbd17ba7dee16a1" } modify_external_catalogs: true
I20250629 01:51:45.501794 738 tablet_service.cc:1468] Processing CreateTablet for tablet 8295e8178eff4d5eb95ac885092731e0 (DEFAULT_TABLE table=default.Table_0d8a47c941e64eb1bd5332c0dbee8504 [id=c17c3600bb2447d69097c8223ce973f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.502146 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8295e8178eff4d5eb95ac885092731e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.505409 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5f98a84fe7341ac8e6b11b04066323c" } modify_external_catalogs: true
I20250629 01:51:45.505563 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5f98a84fe7341ac8e6b11b04066323c" } modify_external_catalogs: true
I20250629 01:51:45.507206 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a403c9cdeab14d55a3defb5ae2056216"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.507647 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a403c9cdeab14d55a3defb5ae2056216 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:45.508948 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9cfff25bea764e62954e49611b71703c consensus state has no leader
W20250629 01:51:45.509481 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26d85fd3d3dd4b07a812c019e1a5c748 consensus state has no leader
I20250629 01:51:45.510476 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_55cf597f7eab480299bb4d32ccf5868c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.510900 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55cf597f7eab480299bb4d32ccf5868c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.512867 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ea26ca313df64a08bdf66b12c7b74613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.513245 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea26ca313df64a08bdf66b12c7b74613 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:45.516301 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 258994dd24804f4aaa5e299436fac9b0 consensus state has no leader
I20250629 01:51:45.521221 745 tablet_service.cc:1468] Processing CreateTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 (DEFAULT_TABLE table=default.Table_af4bfaa5668e4c499de2890f2792c9eb [id=64e6f3a1044a485dac278b2deccc979d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.521548 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d6f1eb43b24fda94aa0afcf4501195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.521859 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 05905638c6e241a095438f277f53adab (table default.Table_40038832d9f84a35b7db42595981b08d [id=151987e142c54e60b5137314e329dd82]) successfully deleted
I20250629 01:51:45.526445 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c2118c2a7b06486fabe69e25ca0837f4
I20250629 01:51:45.526778 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2b15ded2e96e4eb5a060fa718fc40116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.527154 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b15ded2e96e4eb5a060fa718fc40116 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.528285 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:45.528905 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:45.529783 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 819ed44d98f14075b9a67b88dc44e86e
W20250629 01:51:45.532215 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 327c303b6b9347f5b4cb78f74d41a44a consensus state has no leader
I20250629 01:51:45.533231 745 tablet_service.cc:1468] Processing CreateTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a (DEFAULT_TABLE table=default.Table_4c27cb96dae6400b892abad969dbeecb [id=d62355a430724b96abb294c451c9bd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.533540 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 327c303b6b9347f5b4cb78f74d41a44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.535534 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a (DEFAULT_TABLE table=default.Table_4c27cb96dae6400b892abad969dbeecb [id=d62355a430724b96abb294c451c9bd58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.536132 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 327c303b6b9347f5b4cb78f74d41a44a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.537098 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_657df5e298e548eab160b135980fba3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.537493 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_657df5e298e548eab160b135980fba3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.538195 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b59e9b7111bd4bbbbc21c3c8119cc42d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.538583 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b59e9b7111bd4bbbbc21c3c8119cc42d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.540835 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_57b8fd5efba54e2086b5e33f52312b34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.541332 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3ca4f32bc0954a7c92345faedf91b471" } modify_external_catalogs: true
W20250629 01:51:45.541427 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57b8fd5efba54e2086b5e33f52312b34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.541538 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3ca4f32bc0954a7c92345faedf91b471" } modify_external_catalogs: true
I20250629 01:51:45.542496 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af4bfaa5668e4c499de2890f2792c9eb" } new_table_name: "default.Table_b1a50d6abfb64bf5a297be7fc5c93376" modify_external_catalogs: true
I20250629 01:51:45.547514 892 tablet_service.cc:1515] Processing DeleteTablet for tablet dfee69f9484046d8abe10d2f07033e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.547545 1548 raft_consensus.cc:491] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.548861 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet dd709b557b334434a6c2c4a60a996342
I20250629 01:51:45.549453 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.549816 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.551865 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.553709 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.553802 756 tablet_service.cc:1468] Processing CreateTablet for tablet 303f77a729c648f9b1146d11056b242d (DEFAULT_TABLE table=default.Table_9914b6019e564cf7aa3ad9c511898fed [id=7c4d6f0956f142d195bd44428072be8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.553952 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 819ed44d98f14075b9a67b88dc44e86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.554096 751 tablet_service.cc:1468] Processing CreateTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f (DEFAULT_TABLE table=default.Table_93247e23895942578dd7e5f95a2cd9e4 [id=2ba8a990fc3a474aac09caa7380ccb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.554181 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303f77a729c648f9b1146d11056b242d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.554421 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.554127 747 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9d2aa33a5c4d9ebecaa83c32b39dcb (DEFAULT_TABLE table=default.Table_d1f5b20a8d5d4c7687aedcb1c9e54c2c [id=7a043339924b4d5a814f8aa156943aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.555300 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9d2aa33a5c4d9ebecaa83c32b39dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.556337 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 303f77a729c648f9b1146d11056b242d (DEFAULT_TABLE table=default.Table_9914b6019e564cf7aa3ad9c511898fed [id=7c4d6f0956f142d195bd44428072be8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.556654 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303f77a729c648f9b1146d11056b242d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.560204 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.560391 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 819ed44d98f14075b9a67b88dc44e86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.555995 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9d2aa33a5c4d9ebecaa83c32b39dcb (DEFAULT_TABLE table=default.Table_d1f5b20a8d5d4c7687aedcb1c9e54c2c [id=7a043339924b4d5a814f8aa156943aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.560731 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9d2aa33a5c4d9ebecaa83c32b39dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.561528 1154 tablet_bootstrap.cc:492] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.561654 1154 ts_tablet_manager.cc:1397] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.130s user 0.005s sys 0.004s
I20250629 01:51:45.563103 1154 raft_consensus.cc:357] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.563182 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 819ed44d98f14075b9a67b88dc44e86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.563264 1154 raft_consensus.cc:383] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.563493 1154 raft_consensus.cc:738] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.563710 1154 consensus_queue.cc:260] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.564379 1154 ts_tablet_manager.cc:1428] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250629 01:51:45.564615 1154 tablet_bootstrap.cc:492] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.564527 1548 raft_consensus.cc:513] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.566286 756 tablet_service.cc:1515] Processing DeleteTablet for tablet dd709b557b334434a6c2c4a60a996342 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.566594 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:45.572502 1466 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4712643b602e4f898752fcab6a1e5913" instance_seqno: 1751161898562798) as {username='slave'} at 127.0.104.3:41315; Asking this server to re-register.
W20250629 01:51:45.572865 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 196f1bd233ab4399a9d22a55b8bc2635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 196f1bd233ab4399a9d22a55b8bc2635 already in progress: creating tablet
I20250629 01:51:45.573318 1097 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:45.573630 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7748846a3df14ffa801321ccde86e35f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:45.574347 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet dd709b557b334434a6c2c4a60a996342 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.572640 877 tablet_service.cc:1515] Processing DeleteTablet for tablet dd709b557b334434a6c2c4a60a996342 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:45.574990 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 819ed44d98f14075b9a67b88dc44e86e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 819ed44d98f14075b9a67b88dc44e86e already in progress: creating tablet
I20250629 01:51:45.575162 1548 leader_election.cc:290] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:45.575522 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet dd709b557b334434a6c2c4a60a996342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd709b557b334434a6c2c4a60a996342 already in progress: creating tablet
W20250629 01:51:45.580693 850 leader_election.cc:343] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.581490 1155 tablet_bootstrap.cc:492] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:45.581609 1155 ts_tablet_manager.cc:1397] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.225s user 0.012s sys 0.030s
I20250629 01:51:45.582294 1155 raft_consensus.cc:357] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.582453 1155 raft_consensus.cc:383] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.582542 1155 raft_consensus.cc:738] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.582738 1155 consensus_queue.cc:260] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.587026 1466 ts_manager.cc:194] Registered new tserver with Master: 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.588971 1154 tablet_bootstrap.cc:654] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:45.589780 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4d6f1eb43b24fda94aa0afcf4501195 consensus state has no leader
I20250629 01:51:45.599475 753 tablet_service.cc:1468] Processing CreateTablet for tablet a12fffe61d7140308acd181bfa3d2700 (DEFAULT_TABLE table=default.Table_223fe61dd1034f42a871c3ff477bcf1e [id=075cbade2bd84a68a76f354b7796c991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.600082 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12fffe61d7140308acd181bfa3d2700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.601504 1155 ts_tablet_manager.cc:1428] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250629 01:51:45.602948 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7748846a3df14ffa801321ccde86e35f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:45.605844 850 leader_election.cc:343] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.605953 850 leader_election.cc:304] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:45.606427 1548 raft_consensus.cc:2747] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:45.606868 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 575006c12f7c41658e234f7c6c425d69 with delete_type TABLET_DATA_DELETED (replaced by tablet a792a26725954adea15ad57e31bfb224) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.607122 1156 tablet_replica.cc:331] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.607283 1156 raft_consensus.cc:2241] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.607429 1156 raft_consensus.cc:2270] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.607690 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.608417 1156 ts_tablet_manager.cc:1905] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.614122 1154 tablet_bootstrap.cc:492] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.614300 1154 ts_tablet_manager.cc:1397] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.012s
I20250629 01:51:45.615267 1154 raft_consensus.cc:357] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.615495 1154 raft_consensus.cc:383] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.615576 1154 raft_consensus.cc:738] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.615919 1154 consensus_queue.cc:260] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.616518 1154 ts_tablet_manager.cc:1428] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:45.616966 1154 tablet_bootstrap.cc:492] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.618865 1154 tablet_bootstrap.cc:654] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.620429 1501 raft_consensus.cc:1215] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.564] Dedup: 1.90->[1.91-1.564]
I20250629 01:51:45.621289 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 with delete_type TABLET_DATA_DELETED (replaced by tablet e82b1352bb4d49769808dc402c85890f) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.621428 1156 ts_tablet_manager.cc:1918] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.621781 1155 tablet_bootstrap.cc:492] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:45.627262 1156 log.cc:1199] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/575006c12f7c41658e234f7c6c425d69
I20250629 01:51:45.628187 1156 ts_tablet_manager.cc:1939] T 575006c12f7c41658e234f7c6c425d69 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:45.630761 1155 tablet_bootstrap.cc:654] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.630856 1158 ts_tablet_manager.cc:1939] T 575006c12f7c41658e234f7c6c425d69 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.631757 1158 tablet_replica.cc:331] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.631943 1158 raft_consensus.cc:2241] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.632169 1158 raft_consensus.cc:2270] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.633464 1158 ts_tablet_manager.cc:1905] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.636797 877 tablet_service.cc:1468] Processing CreateTablet for tablet 7908afa2abaa4521b44f954e0790f986 (DEFAULT_TABLE table=default.Table_ecd89721aae04a9abed2f4ff44cfd3dd [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.638029 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7908afa2abaa4521b44f954e0790f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.639110 1158 ts_tablet_manager.cc:1918] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.639201 1158 log.cc:1199] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/648fd1ca8f514ea3ba4554452b19933f
I20250629 01:51:45.639472 1158 ts_tablet_manager.cc:1939] T 648fd1ca8f514ea3ba4554452b19933f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.640061 880 tablet_service.cc:1468] Processing CreateTablet for tablet 07e0727102d54294a7c13a4aeef311e1 (DEFAULT_TABLE table=default.Table_a9fddce63afb42b4be73a8c4807e050a [id=d8e51d2596fa4215bd2e2a3c43c466f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.640525 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4030891de41444e0b387492e570a11a5 (DEFAULT_TABLE table=default.Table_fceee991e4084064b9ec489915a42d1e [id=6f5c85c18e9e4fc4b9f26bc1e5c11690]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.643445 1155 tablet_bootstrap.cc:492] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:45.643555 886 tablet_service.cc:1468] Processing CreateTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 (DEFAULT_TABLE table=default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.643626 1155 ts_tablet_manager.cc:1397] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20250629 01:51:45.644316 889 tablet_service.cc:1468] Processing CreateTablet for tablet e5d27d658892496a9b19b57949a8399a (DEFAULT_TABLE table=default.Table_768dd2c93bfa4a828037ed23b11dc5ae [id=e8832475ef554a2ebbf61f3481bf197c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.644331 1155 raft_consensus.cc:357] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.644546 1155 raft_consensus.cc:383] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.644654 1155 raft_consensus.cc:738] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.644914 878 tablet_service.cc:1468] Processing CreateTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd (DEFAULT_TABLE table=default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.644915 1155 consensus_queue.cc:260] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.643927 883 tablet_service.cc:1468] Processing CreateTablet for tablet 50d23f589de84dd5b150a17cee00c411 (DEFAULT_TABLE table=default.Table_56ce186b15474e3ebf35ac8f454778bf [id=5a9ebb3e7cd54996a1d7e312bcc3d8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.645581 893 tablet_service.cc:1468] Processing CreateTablet for tablet d062fb830020442faee626be006926c1 (DEFAULT_TABLE table=default.Table_06317866df174e2db51b073ad43db5db [id=3ab3a1e396af475788b1dd132f662f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.647905 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e0727102d54294a7c13a4aeef311e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.648113 1155 ts_tablet_manager.cc:1428] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:45.649374 895 tablet_service.cc:1468] Processing CreateTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b (DEFAULT_TABLE table=default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.650040 890 tablet_service.cc:1468] Processing CreateTablet for tablet 76118a6e684448b09f244d44c909ccaa (DEFAULT_TABLE table=default.Table_be093346e01e4a688e5ede68f9e96fbd [id=dcc07a46945a4add8e667bfea51dfed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.650254 1155 tablet_bootstrap.cc:492] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:45.650357 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d062fb830020442faee626be006926c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.651587 1154 tablet_bootstrap.cc:492] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.651746 1154 ts_tablet_manager.cc:1397] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.004s
I20250629 01:51:45.652570 1155 tablet_bootstrap.cc:654] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.652879 1154 raft_consensus.cc:357] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.653115 1154 raft_consensus.cc:383] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.653347 1158 tablet_replica.cc:331] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.653388 1154 raft_consensus.cc:738] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.653606 1158 raft_consensus.cc:2241] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.653822 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.654436 1158 raft_consensus.cc:2270] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.653676 1154 consensus_queue.cc:260] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.655220 1158 ts_tablet_manager.cc:1905] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.655867 1154 ts_tablet_manager.cc:1428] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250629 01:51:45.656149 1154 tablet_bootstrap.cc:492] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.657481 1154 tablet_bootstrap.cc:654] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.660188 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.676175 1157 tablet_replica.cc:331] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.677418 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.677798 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e5d27d658892496a9b19b57949a8399a (DEFAULT_TABLE table=default.Table_768dd2c93bfa4a828037ed23b11dc5ae [id=e8832475ef554a2ebbf61f3481bf197c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.678120 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d27d658892496a9b19b57949a8399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.678265 738 tablet_service.cc:1468] Processing CreateTablet for tablet 50d23f589de84dd5b150a17cee00c411 (DEFAULT_TABLE table=default.Table_56ce186b15474e3ebf35ac8f454778bf [id=5a9ebb3e7cd54996a1d7e312bcc3d8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.678586 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d23f589de84dd5b150a17cee00c411. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.678779 1159 tablet_bootstrap.cc:492] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.679224 1159 ts_tablet_manager.cc:1397] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.232s user 0.037s sys 0.066s
I20250629 01:51:45.680392 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 07e0727102d54294a7c13a4aeef311e1 (DEFAULT_TABLE table=default.Table_a9fddce63afb42b4be73a8c4807e050a [id=d8e51d2596fa4215bd2e2a3c43c466f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.680712 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e0727102d54294a7c13a4aeef311e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.681821 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76118a6e684448b09f244d44c909ccaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.683944 1158 ts_tablet_manager.cc:1918] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.684834 756 tablet_service.cc:1468] Processing CreateTablet for tablet 4030891de41444e0b387492e570a11a5 (DEFAULT_TABLE table=default.Table_fceee991e4084064b9ec489915a42d1e [id=6f5c85c18e9e4fc4b9f26bc1e5c11690]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.685163 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4030891de41444e0b387492e570a11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.685597 1157 raft_consensus.cc:2241] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.688359 751 tablet_service.cc:1468] Processing CreateTablet for tablet e5d27d658892496a9b19b57949a8399a (DEFAULT_TABLE table=default.Table_768dd2c93bfa4a828037ed23b11dc5ae [id=e8832475ef554a2ebbf61f3481bf197c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.688675 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d27d658892496a9b19b57949a8399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.684082 1158 log.cc:1199] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a1938e67acaf4ab19fcbed13e1f1bfc6
I20250629 01:51:45.689168 1158 ts_tablet_manager.cc:1939] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.689957 1158 tablet_replica.cc:331] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.690111 1158 raft_consensus.cc:2241] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.690254 1158 raft_consensus.cc:2270] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.690340 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4030891de41444e0b387492e570a11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.691272 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 575006c12f7c41658e234f7c6c425d69 (table default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]) successfully deleted
I20250629 01:51:45.691464 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 648fd1ca8f514ea3ba4554452b19933f (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]) successfully deleted
I20250629 01:51:45.691546 1158 ts_tablet_manager.cc:1905] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:45.691663 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 196f1bd233ab4399a9d22a55b8bc2635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 196f1bd233ab4399a9d22a55b8bc2635 already in progress: creating tablet
I20250629 01:51:45.691727 747 tablet_service.cc:1468] Processing CreateTablet for tablet 76118a6e684448b09f244d44c909ccaa (DEFAULT_TABLE table=default.Table_be093346e01e4a688e5ede68f9e96fbd [id=dcc07a46945a4add8e667bfea51dfed7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:45.691843 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 819ed44d98f14075b9a67b88dc44e86e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 819ed44d98f14075b9a67b88dc44e86e already in progress: creating tablet
W20250629 01:51:45.692023 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet dd709b557b334434a6c2c4a60a996342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd709b557b334434a6c2c4a60a996342 already in progress: creating tablet
I20250629 01:51:45.692170 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a1938e67acaf4ab19fcbed13e1f1bfc6 (table default.Table_38d08e8c1c0d4beea36513f66e9cbe43 [id=e12703cc30a94230b4c5f0fc21988359]) successfully deleted
I20250629 01:51:45.692327 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 575006c12f7c41658e234f7c6c425d69 (table default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]) successfully deleted
I20250629 01:51:45.694234 1548 raft_consensus.cc:491] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.694690 1157 raft_consensus.cc:2270] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.695000 1548 raft_consensus.cc:513] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.695050 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 76118a6e684448b09f244d44c909ccaa (DEFAULT_TABLE table=default.Table_be093346e01e4a688e5ede68f9e96fbd [id=dcc07a46945a4add8e667bfea51dfed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.695385 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a12fffe61d7140308acd181bfa3d2700 (DEFAULT_TABLE table=default.Table_223fe61dd1034f42a871c3ff477bcf1e [id=075cbade2bd84a68a76f354b7796c991]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.695892 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76118a6e684448b09f244d44c909ccaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.696116 1157 ts_tablet_manager.cc:1905] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.696309 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a12fffe61d7140308acd181bfa3d2700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.696501 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 50d23f589de84dd5b150a17cee00c411 (DEFAULT_TABLE table=default.Table_56ce186b15474e3ebf35ac8f454778bf [id=5a9ebb3e7cd54996a1d7e312bcc3d8fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.696813 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d23f589de84dd5b150a17cee00c411. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.697221 1548 leader_election.cc:290] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:45.697317 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d27d658892496a9b19b57949a8399a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.698666 1159 raft_consensus.cc:357] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.699438 1159 raft_consensus.cc:383] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.699522 1159 raft_consensus.cc:738] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.684227 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b (DEFAULT_TABLE table=default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.700423 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ab0aa4e74d345b2a214b90b5018c82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.700467 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f (DEFAULT_TABLE table=default.Table_93247e23895942578dd7e5f95a2cd9e4 [id=2ba8a990fc3a474aac09caa7380ccb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.700769 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.700906 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfee69f9484046d8abe10d2f07033e09" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:45.701368 850 leader_election.cc:343] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.684413 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7908afa2abaa4521b44f954e0790f986 (DEFAULT_TABLE table=default.Table_ecd89721aae04a9abed2f4ff44cfd3dd [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.684594 1015 tablet_service.cc:1468] Processing CreateTablet for tablet d062fb830020442faee626be006926c1 (DEFAULT_TABLE table=default.Table_06317866df174e2db51b073ad43db5db [id=3ab3a1e396af475788b1dd132f662f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.683897 1031 tablet_service.cc:1468] Processing CreateTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 (DEFAULT_TABLE table=default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.705713 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7908afa2abaa4521b44f954e0790f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.708252 1159 consensus_queue.cc:260] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.709774 1159 ts_tablet_manager.cc:1428] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20250629 01:51:45.710054 1159 tablet_bootstrap.cc:492] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:45.711831 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 5e9f6de505a34afbae48a373b4b2a0cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.712105 739 tablet_service.cc:1468] Processing CreateTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b (DEFAULT_TABLE table=default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.712334 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9a4eca53d6d5404fa56df4720474d29c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.712426 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dfee69f9484046d8abe10d2f07033e09" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:45.712818 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a4eca53d6d5404fa56df4720474d29c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.713070 1156 tablet_replica.cc:331] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:45.713238 1156 raft_consensus.cc:2241] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.713430 757 tablet_service.cc:1468] Processing CreateTablet for tablet d062fb830020442faee626be006926c1 (DEFAULT_TABLE table=default.Table_06317866df174e2db51b073ad43db5db [id=3ab3a1e396af475788b1dd132f662f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.713541 740 tablet_service.cc:1468] Processing CreateTablet for tablet 7908afa2abaa4521b44f954e0790f986 (DEFAULT_TABLE table=default.Table_ecd89721aae04a9abed2f4ff44cfd3dd [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.713667 755 tablet_service.cc:1468] Processing CreateTablet for tablet 07e0727102d54294a7c13a4aeef311e1 (DEFAULT_TABLE table=default.Table_a9fddce63afb42b4be73a8c4807e050a [id=d8e51d2596fa4215bd2e2a3c43c466f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.713835 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76118a6e684448b09f244d44c909ccaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.714628 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_50eb1d5f0c5f41e0a975713cf15124ec" } new_table_name: "default.Table_88b58172524f46d7bfd7bbc16ec68943" modify_external_catalogs: true
I20250629 01:51:45.715165 1156 raft_consensus.cc:2270] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.716570 754 tablet_service.cc:1468] Processing CreateTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 (DEFAULT_TABLE table=default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.717708 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5bb5e8cd427a4fdf9a1ccbc0fb4caaf4" } new_table_name: "default.Table_0a60f2cb55074b998ddf4c8a1c30ea84" modify_external_catalogs: true
I20250629 01:51:45.718348 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7fec8ba2ba174536baceb16beaa50ab8" } new_table_name: "default.Table_cdb3a884c9ca4f72803e5bc54a78d34a" modify_external_catalogs: true
I20250629 01:51:45.719763 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ab0aa4e74d345b2a214b90b5018c82b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.720041 850 leader_election.cc:343] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.721961 1156 ts_tablet_manager.cc:1905] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.723134 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7908afa2abaa4521b44f954e0790f986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.724028 850 leader_election.cc:304] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:45.724632 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d062fb830020442faee626be006926c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.725359 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd (DEFAULT_TABLE table=default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.725415 1547 raft_consensus.cc:2747] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:45.727171 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d062fb830020442faee626be006926c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.728382 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd (DEFAULT_TABLE table=default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.728776 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.728705 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ab0aa4e74d345b2a214b90b5018c82b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.730127 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_abc4bc33793d4bee824f97f24a172593" } modify_external_catalogs: true
I20250629 01:51:45.723655 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_48729fdc20bf43259c31574125ecaa17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.730263 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_abc4bc33793d4bee824f97f24a172593" } modify_external_catalogs: true
I20250629 01:51:45.731143 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0d8a47c941e64eb1bd5332c0dbee8504" } modify_external_catalogs: true
I20250629 01:51:45.731281 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0d8a47c941e64eb1bd5332c0dbee8504" } modify_external_catalogs: true
W20250629 01:51:45.732475 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48729fdc20bf43259c31574125ecaa17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.734403 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50d23f589de84dd5b150a17cee00c411. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.735056 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:45.740684 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5c353375f2c64ac39356062ccc4a5986" } new_table_name: "default.Table_5149d5ed90ca4d6ca10f30e1694a0e1d" modify_external_catalogs: true
W20250629 01:51:45.742740 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b55c66136ba43d7b535750e04b72308 consensus state has no leader
W20250629 01:51:45.743216 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf4b52fae47f45ad9d77b0a97a854860 consensus state has no leader
I20250629 01:51:45.743633 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:45.744351 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_04cc9d29a25645f0b3facf31bd7a9f57" } modify_external_catalogs: true
I20250629 01:51:45.744503 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_04cc9d29a25645f0b3facf31bd7a9f57" } modify_external_catalogs: true
W20250629 01:51:45.749037 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bb4d7c2096a45df8eb392a2e7d85c3d consensus state has no leader
I20250629 01:51:45.750188 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8295e8178eff4d5eb95ac885092731e0
I20250629 01:51:45.752080 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.752631 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 8295e8178eff4d5eb95ac885092731e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.752830 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.753196 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2df86f12bdf34d789bea625355a339a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.754168 738 tablet_service.cc:1515] Processing DeleteTablet for tablet df66984acfa44a99bf69096aff6d49ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.754321 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07e0727102d54294a7c13a4aeef311e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.756067 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2df86f12bdf34d789bea625355a339a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.756525 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_30b8e1c0bc8241a7ba265c3c4f86a095"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.756978 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30b8e1c0bc8241a7ba265c3c4f86a095 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.757805 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a968452543bd46baa06280c24819ada1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.758219 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a968452543bd46baa06280c24819ada1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.758754 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.760974 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.760852 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2" } modify_external_catalogs: true
I20250629 01:51:45.761299 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2" } modify_external_catalogs: true
W20250629 01:51:45.762635 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56b9ef027c9d42048e4a6dff221df793 consensus state has no leader
I20250629 01:51:45.763185 893 tablet_service.cc:1515] Processing DeleteTablet for tablet df66984acfa44a99bf69096aff6d49ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.764142 1155 tablet_bootstrap.cc:492] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:45.764279 1155 ts_tablet_manager.cc:1397] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.009s
I20250629 01:51:45.764981 1155 raft_consensus.cc:357] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.765146 1155 raft_consensus.cc:383] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.765233 1155 raft_consensus.cc:738] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.765937 1154 tablet_bootstrap.cc:492] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.766438 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.767097 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.768664 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8295e8178eff4d5eb95ac885092731e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.769971 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet df66984acfa44a99bf69096aff6d49ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.772328 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 4030891de41444e0b387492e570a11a5 (DEFAULT_TABLE table=default.Table_fceee991e4084064b9ec489915a42d1e [id=6f5c85c18e9e4fc4b9f26bc1e5c11690]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.773299 1154 ts_tablet_manager.cc:1397] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.110s user 0.007s sys 0.004s
I20250629 01:51:45.773286 1547 raft_consensus.cc:491] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.774017 1154 raft_consensus.cc:357] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.781049 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 683c87ea2ad44cecb2366d4a846d9195
I20250629 01:51:45.783598 748 tablet_service.cc:1468] Processing CreateTablet for tablet c62678645ff54aa6b3809212f4d66ec2 (DEFAULT_TABLE table=default.Table_b59e9b7111bd4bbbbc21c3c8119cc42d [id=e535358947de4a289f0bdbf8c3b1f488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.786427 741 tablet_service.cc:1468] Processing CreateTablet for tablet 3ba5887dffc1460a89208447ecc6a198 (DEFAULT_TABLE table=default.Table_abdf20a022384ea199a07cc08b8b9f97 [id=7663b74fe2374460b050e5b3fc56ad49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.786725 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba5887dffc1460a89208447ecc6a198. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.789224 1159 tablet_bootstrap.cc:654] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.789284 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0ed0327b20c3440b9f184793c8e1ee7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.789481 1157 ts_tablet_manager.cc:1918] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:45.789753 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ed0327b20c3440b9f184793c8e1ee7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.791164 1157 log.cc:1199] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/575006c12f7c41658e234f7c6c425d69
I20250629 01:51:45.791491 1157 ts_tablet_manager.cc:1939] T 575006c12f7c41658e234f7c6c425d69 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:45.789391 1547 raft_consensus.cc:513] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.792037 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 8295e8178eff4d5eb95ac885092731e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.792315 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 390) took 1097 ms. Trace:
I20250629 01:51:45.792392 1157 rpcz_store.cc:276] 0629 01:51:44.694179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.884183 (+190004us) service_pool.cc:225] Handling call
0629 01:51:45.521551 (+637368us) ts_tablet_manager.cc:1145] Deleting tablet 575006c12f7c41658e234f7c6c425d69
0629 01:51:45.521559 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.706802 (+185243us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.784180 (+ 77378us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:45.791728 (+ 7548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":637152}
I20250629 01:51:45.793220 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_88b58172524f46d7bfd7bbc16ec68943" } new_table_name: "default.Table_e99d4af37fd7473394598f4c41353fd9" modify_external_catalogs: true
I20250629 01:51:45.793627 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet fbcae02d893f4795a12d26af57b18b85
I20250629 01:51:45.794175 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4030891de41444e0b387492e570a11a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.796360 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_54de7744e36d4183a8af975cdc699bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.796790 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54de7744e36d4183a8af975cdc699bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.796926 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_40c6d676eac2406887e3b761d5357587"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.797348 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40c6d676eac2406887e3b761d5357587 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.797629 746 tablet_service.cc:1468] Processing CreateTablet for tablet dc841eb171d04b118b1e705ceed4c861 (DEFAULT_TABLE table=default.Table_7f95c64f736b4aee86abc0c7785ab8d6 [id=0429a74c16854b3092e37a247f33c497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.798436 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.798780 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_96a279d58b0a48b19a894d999572a340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.798871 749 tablet_service.cc:1468] Processing CreateTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 (DEFAULT_TABLE table=default.Table_2b15ded2e96e4eb5a060fa718fc40116 [id=4a83041e3c7349f7a0b628c27e3a2139]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:45.799191 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96a279d58b0a48b19a894d999572a340 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.799193 743 tablet_service.cc:1468] Processing CreateTablet for tablet 43b845c3e7174618bd5c394d7206a634 (DEFAULT_TABLE table=default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9 [id=ad6049db2ac64d5c86e06c58531e2120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.800380 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62678645ff54aa6b3809212f4d66ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.800710 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b845c3e7174618bd5c394d7206a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.801245 893 tablet_service.cc:1468] Processing CreateTablet for tablet c62678645ff54aa6b3809212f4d66ec2 (DEFAULT_TABLE table=default.Table_b59e9b7111bd4bbbbc21c3c8119cc42d [id=e535358947de4a289f0bdbf8c3b1f488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.802428 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.800856 752 tablet_service.cc:1468] Processing CreateTablet for tablet a582d4f907f340179024c0377d3893e2 (DEFAULT_TABLE table=default.Table_ea26ca313df64a08bdf66b12c7b74613 [id=060d68b1f054440ea9f6e7977adcb982]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:45.804338 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf4b52fae47f45ad9d77b0a97a854860 consensus state has no leader
I20250629 01:51:45.808051 878 tablet_service.cc:1468] Processing CreateTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf (DEFAULT_TABLE table=default.Table_55cf597f7eab480299bb4d32ccf5868c [id=fb540507d98341b2801c8394decec635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.808359 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62678645ff54aa6b3809212f4d66ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.809468 1157 tablet_replica.cc:331] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:45.809700 1547 leader_election.cc:290] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:45.810586 877 tablet_service.cc:1468] Processing CreateTablet for tablet 3ba5887dffc1460a89208447ecc6a198 (DEFAULT_TABLE table=default.Table_abdf20a022384ea199a07cc08b8b9f97 [id=7663b74fe2374460b050e5b3fc56ad49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.811569 1157 raft_consensus.cc:2241] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.812278 1157 raft_consensus.cc:2270] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.812467 750 tablet_service.cc:1468] Processing CreateTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf (DEFAULT_TABLE table=default.Table_55cf597f7eab480299bb4d32ccf5868c [id=fb540507d98341b2801c8394decec635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.812778 742 tablet_service.cc:1468] Processing CreateTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 (DEFAULT_TABLE table=default.Table_a403c9cdeab14d55a3defb5ae2056216 [id=b4d82748e8804c78a1bd48cef02e98a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.812997 744 tablet_service.cc:1468] Processing CreateTablet for tablet 898b49812ece4b57a1a1015d9f26be2f (DEFAULT_TABLE table=default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.813398 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc841eb171d04b118b1e705ceed4c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.802663 738 tablet_service.cc:1468] Processing CreateTablet for tablet c5d45854191247bf8172d0bbfd01b6db (DEFAULT_TABLE table=default.Table_657df5e298e548eab160b135980fba3a [id=7f7fa7fcbfc248319d2a0b8085312cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.815337 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689adaf0c7564d9ba0b31061e4e2d8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.819904 1154 raft_consensus.cc:383] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.820022 1154 raft_consensus.cc:738] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.820344 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a582d4f907f340179024c0377d3893e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.821135 1154 consensus_queue.cc:260] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.824520 1154 ts_tablet_manager.cc:1428] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.048s user 0.001s sys 0.000s
I20250629 01:51:45.826164 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476a6cf28db04f67bc23ddfc5cacad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.827442 1017 tablet_service.cc:1468] Processing CreateTablet for tablet dc841eb171d04b118b1e705ceed4c861 (DEFAULT_TABLE table=default.Table_7f95c64f736b4aee86abc0c7785ab8d6 [id=0429a74c16854b3092e37a247f33c497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.827643 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 (DEFAULT_TABLE table=default.Table_a403c9cdeab14d55a3defb5ae2056216 [id=b4d82748e8804c78a1bd48cef02e98a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.832069 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898b49812ece4b57a1a1015d9f26be2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.832432 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 3ba5887dffc1460a89208447ecc6a198 (DEFAULT_TABLE table=default.Table_abdf20a022384ea199a07cc08b8b9f97 [id=7663b74fe2374460b050e5b3fc56ad49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.833122 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d45854191247bf8172d0bbfd01b6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.836264 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0ca04e5f6b42df8567deb53a3e5f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.836934 747 tablet_service.cc:1515] Processing DeleteTablet for tablet fbcae02d893f4795a12d26af57b18b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:45.840495 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c5d45854191247bf8172d0bbfd01b6db (DEFAULT_TABLE table=default.Table_657df5e298e548eab160b135980fba3a [id=7f7fa7fcbfc248319d2a0b8085312cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.845729 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7f38778518cf47bca0a3dc0dc8e94313"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:45.846199 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f38778518cf47bca0a3dc0dc8e94313 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.851281 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c62678645ff54aa6b3809212f4d66ec2 (DEFAULT_TABLE table=default.Table_b59e9b7111bd4bbbbc21c3c8119cc42d [id=e535358947de4a289f0bdbf8c3b1f488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.851323 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 898b49812ece4b57a1a1015d9f26be2f (DEFAULT_TABLE table=default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.853669 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc841eb171d04b118b1e705ceed4c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.854012 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 43b845c3e7174618bd5c394d7206a634 (DEFAULT_TABLE table=default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9 [id=ad6049db2ac64d5c86e06c58531e2120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.854934 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a582d4f907f340179024c0377d3893e2 (DEFAULT_TABLE table=default.Table_ea26ca313df64a08bdf66b12c7b74613 [id=060d68b1f054440ea9f6e7977adcb982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.856621 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf (DEFAULT_TABLE table=default.Table_55cf597f7eab480299bb4d32ccf5868c [id=fb540507d98341b2801c8394decec635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.857641 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 (DEFAULT_TABLE table=default.Table_2b15ded2e96e4eb5a060fa718fc40116 [id=4a83041e3c7349f7a0b628c27e3a2139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.860623 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476a6cf28db04f67bc23ddfc5cacad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.860713 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7748846a3df14ffa801321ccde86e35f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.862248 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898b49812ece4b57a1a1015d9f26be2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.863272 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0ca04e5f6b42df8567deb53a3e5f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.863811 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689adaf0c7564d9ba0b31061e4e2d8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.864070 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b845c3e7174618bd5c394d7206a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.864256 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a582d4f907f340179024c0377d3893e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.864208 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet fbcae02d893f4795a12d26af57b18b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.865324 1158 ts_tablet_manager.cc:1918] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.865442 1158 log.cc:1199] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/829c64e2c71347f4af2c8870cdd8af31
I20250629 01:51:45.864900 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:45.866398 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62678645ff54aa6b3809212f4d66ec2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.873267 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet cf4b52fae47f45ad9d77b0a97a854860 consensus state has no leader
I20250629 01:51:45.876159 1158 ts_tablet_manager.cc:1939] T 829c64e2c71347f4af2c8870cdd8af31 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.880024 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba5887dffc1460a89208447ecc6a198. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.881489 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d (DEFAULT_TABLE table=default.Table_30b8e1c0bc8241a7ba265c3c4f86a095 [id=5ad43132e07d4ddeb9d1716e8e314256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.882988 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d7e2c70221e94923b6a324a03f7133c6 (DEFAULT_TABLE table=default.Table_48729fdc20bf43259c31574125ecaa17 [id=15ec8050411e48ab97a88c2b363a8ed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.883718 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 8635903caac9480184c32e6689870485 (DEFAULT_TABLE table=default.Table_2df86f12bdf34d789bea625355a339a1 [id=45a7f84c7488427082dd8b4c0b7d6119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.884622 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d45854191247bf8172d0bbfd01b6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.885210 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 (DEFAULT_TABLE table=default.Table_a968452543bd46baa06280c24819ada1 [id=a9ff1c784a004f5cad1cf1a1c8066d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.885783 1154 tablet_bootstrap.cc:492] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.886114 1158 tablet_replica.cc:331] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:45.886497 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8635903caac9480184c32e6689870485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.887360 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 689adaf0c7564d9ba0b31061e4e2d8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.887411 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 829c64e2c71347f4af2c8870cdd8af31 (table default.Table_358f4795c7e244f2b72efe50dacd310f [id=00b7c47fc4684174b514feaab3b7d411]) successfully deleted
W20250629 01:51:45.887637 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 49333302a27f4ac3b76a1e492b3c1b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49333302a27f4ac3b76a1e492b3c1b60 already in progress: creating tablet
W20250629 01:51:45.887804 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8295e8178eff4d5eb95ac885092731e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8295e8178eff4d5eb95ac885092731e0 already in progress: creating tablet
W20250629 01:51:45.888002 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet df66984acfa44a99bf69096aff6d49ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df66984acfa44a99bf69096aff6d49ab already in progress: creating tablet
I20250629 01:51:45.888552 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ba5887dffc1460a89208447ecc6a198. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.888476 889 tablet_service.cc:1468] Processing CreateTablet for tablet c5d45854191247bf8172d0bbfd01b6db (DEFAULT_TABLE table=default.Table_657df5e298e548eab160b135980fba3a [id=7f7fa7fcbfc248319d2a0b8085312cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.889385 1157 ts_tablet_manager.cc:1905] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.889647 1547 raft_consensus.cc:491] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:45.891110 895 tablet_service.cc:1468] Processing CreateTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 (DEFAULT_TABLE table=default.Table_a403c9cdeab14d55a3defb5ae2056216 [id=b4d82748e8804c78a1bd48cef02e98a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.891587 1547 raft_consensus.cc:513] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.892153 1547 leader_election.cc:290] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.893164 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5d45854191247bf8172d0bbfd01b6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.890592 883 tablet_service.cc:1468] Processing CreateTablet for tablet 898b49812ece4b57a1a1015d9f26be2f (DEFAULT_TABLE table=default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.895171 890 tablet_service.cc:1468] Processing CreateTablet for tablet 43b845c3e7174618bd5c394d7206a634 (DEFAULT_TABLE table=default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9 [id=ad6049db2ac64d5c86e06c58531e2120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.895799 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.897359 1158 raft_consensus.cc:2241] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:45.897531 1158 raft_consensus.cc:2270] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:45.898296 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d0fbb691514d51a1ca71f32a666c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.899111 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898b49812ece4b57a1a1015d9f26be2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.900416 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b845c3e7174618bd5c394d7206a634. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.900828 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 476a6cf28db04f67bc23ddfc5cacad87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.900419 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b (DEFAULT_TABLE table=default.Table_9a4eca53d6d5404fa56df4720474d29c [id=e61ee06c5a0343ecb060fea35f151e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.903018 892 tablet_service.cc:1468] Processing CreateTablet for tablet dc841eb171d04b118b1e705ceed4c861 (DEFAULT_TABLE table=default.Table_7f95c64f736b4aee86abc0c7785ab8d6 [id=0429a74c16854b3092e37a247f33c497]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.903709 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e2c70221e94923b6a324a03f7133c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.903734 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc841eb171d04b118b1e705ceed4c861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.905022 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a8a9bc691e4b158276f34a51c0dc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.905706 1158 ts_tablet_manager.cc:1905] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:45.906978 880 tablet_service.cc:1468] Processing CreateTablet for tablet a582d4f907f340179024c0377d3893e2 (DEFAULT_TABLE table=default.Table_ea26ca313df64a08bdf66b12c7b74613 [id=060d68b1f054440ea9f6e7977adcb982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.908617 887 tablet_service.cc:1468] Processing CreateTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 (DEFAULT_TABLE table=default.Table_2b15ded2e96e4eb5a060fa718fc40116 [id=4a83041e3c7349f7a0b628c27e3a2139]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.909616 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.910018 879 tablet_service.cc:1515] Processing DeleteTablet for tablet fbcae02d893f4795a12d26af57b18b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:45.910142 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a582d4f907f340179024c0377d3893e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.917451 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0ca04e5f6b42df8567deb53a3e5f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.920527 891 tablet_service.cc:1468] Processing CreateTablet for tablet d7e2c70221e94923b6a324a03f7133c6 (DEFAULT_TABLE table=default.Table_48729fdc20bf43259c31574125ecaa17 [id=15ec8050411e48ab97a88c2b363a8ed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.920964 879 tablet_service.cc:1468] Processing CreateTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d (DEFAULT_TABLE table=default.Table_30b8e1c0bc8241a7ba265c3c4f86a095 [id=5ad43132e07d4ddeb9d1716e8e314256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.920984 739 tablet_service.cc:1468] Processing CreateTablet for tablet 8635903caac9480184c32e6689870485 (DEFAULT_TABLE table=default.Table_2df86f12bdf34d789bea625355a339a1 [id=45a7f84c7488427082dd8b4c0b7d6119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.921273 896 tablet_service.cc:1468] Processing CreateTablet for tablet 8635903caac9480184c32e6689870485 (DEFAULT_TABLE table=default.Table_2df86f12bdf34d789bea625355a339a1 [id=45a7f84c7488427082dd8b4c0b7d6119]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.921875 882 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b (DEFAULT_TABLE table=default.Table_9a4eca53d6d5404fa56df4720474d29c [id=e61ee06c5a0343ecb060fea35f151e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.922036 885 tablet_service.cc:1468] Processing CreateTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 (DEFAULT_TABLE table=default.Table_a968452543bd46baa06280c24819ada1 [id=a9ff1c784a004f5cad1cf1a1c8066d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.922596 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6128463cd85d4f0cbfe21b0e5bd0e470" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:45.923131 751 tablet_service.cc:1468] Processing CreateTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d (DEFAULT_TABLE table=default.Table_30b8e1c0bc8241a7ba265c3c4f86a095 [id=5ad43132e07d4ddeb9d1716e8e314256]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.923825 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b (DEFAULT_TABLE table=default.Table_9a4eca53d6d5404fa56df4720474d29c [id=e61ee06c5a0343ecb060fea35f151e6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.924072 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e2c70221e94923b6a324a03f7133c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.925382 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6128463cd85d4f0cbfe21b0e5bd0e470" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:45.925721 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d7b8cf118eb4a5c997815b97b9b7d00" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:45.926174 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d7b8cf118eb4a5c997815b97b9b7d00" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:45.926347 1048 raft_consensus.cc:2466] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:45.927935 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.928196 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8635903caac9480184c32e6689870485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.933776 1156 ts_tablet_manager.cc:1918] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.933936 1156 log.cc:1199] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5e9f6de505a34afbae48a373b4b2a0cf
I20250629 01:51:45.934274 1156 ts_tablet_manager.cc:1939] T 5e9f6de505a34afbae48a373b4b2a0cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:45.934563 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d0fbb691514d51a1ca71f32a666c2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.935895 850 leader_election.cc:343] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.936213 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a8a9bc691e4b158276f34a51c0dc61. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:45.937503 850 leader_election.cc:343] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:45.937606 850 leader_election.cc:304] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:45.938258 850 leader_election.cc:304] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:45.939265 1547 raft_consensus.cc:2802] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:45.939347 1547 raft_consensus.cc:491] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:45.939879 1547 raft_consensus.cc:3058] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:45.940058 1548 raft_consensus.cc:2747] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:45.941888 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d0fbb691514d51a1ca71f32a666c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.942384 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8635903caac9480184c32e6689870485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.943042 747 tablet_service.cc:1468] Processing CreateTablet for tablet d7e2c70221e94923b6a324a03f7133c6 (DEFAULT_TABLE table=default.Table_48729fdc20bf43259c31574125ecaa17 [id=15ec8050411e48ab97a88c2b363a8ed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.943564 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.944042 1547 raft_consensus.cc:513] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.956321 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7eb487cf3b984b829ea5456c141ea824" } modify_external_catalogs: true
I20250629 01:51:45.956452 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7b31a72aec3c4710971cd48931f10395" } new_table_name: "default.Table_33ab0761369a4bd280345f65e285d02b" modify_external_catalogs: true
I20250629 01:51:45.956566 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7eb487cf3b984b829ea5456c141ea824" } modify_external_catalogs: true
I20250629 01:51:45.957353 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f5ba36b31e8a415b81ec003950152c56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.957862 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_acc52ee4aee240dc82efdc80f09faa91" } new_table_name: "default.Table_278cd76ddcb64217983536c8069d1803" modify_external_catalogs: true
W20250629 01:51:45.958031 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5ba36b31e8a415b81ec003950152c56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.958079 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_adf0dc244a084dcf82f6cb6f22ecc128" } new_table_name: "default.Table_770438a7f7324802ab8dea0697fc2423" modify_external_catalogs: true
I20250629 01:51:45.958981 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e43be787310e423097abae89ed26eb57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:45.959311 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_208f2046cc46452887e343bab242e4b2" } modify_external_catalogs: true
I20250629 01:51:45.959439 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_208f2046cc46452887e343bab242e4b2" } modify_external_catalogs: true
I20250629 01:51:45.959700 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f641040c5d3f4fabb4642ae6c32b10c7" } modify_external_catalogs: true
I20250629 01:51:45.959821 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f641040c5d3f4fabb4642ae6c32b10c7" } modify_external_catalogs: true
W20250629 01:51:45.959443 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e43be787310e423097abae89ed26eb57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:45.963994 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 575006c12f7c41658e234f7c6c425d69 (table default.Table_e63b2128684f4cfc898dee216184ddfa [id=b7fb18171e21446d99879ee1cb1cc960]) successfully deleted
I20250629 01:51:45.964246 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a9b6ef9e25bf4f8a9e3e8e0f9ce2060e" } new_table_name: "default.Table_f5bc90b9f24640639b11b63857131036" modify_external_catalogs: true
W20250629 01:51:45.964263 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0b16b6840c24484b90218fffc1da9621 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0b16b6840c24484b90218fffc1da9621 already in progress: creating tablet
W20250629 01:51:45.964861 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b9b680859a1a4c56afe33f5b1b3c0793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9b680859a1a4c56afe33f5b1b3c0793 already in progress: creating tablet
W20250629 01:51:45.965073 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 68cedbd093b4420d8f7b29219ce8f77b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68cedbd093b4420d8f7b29219ce8f77b already in progress: creating tablet
I20250629 01:51:45.966080 740 tablet_service.cc:1468] Processing CreateTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 (DEFAULT_TABLE table=default.Table_a968452543bd46baa06280c24819ada1 [id=a9ff1c784a004f5cad1cf1a1c8066d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:45.966696 447 master_service.cc:948] ReplaceTablet: received request to replace tablet a90b206d58d04acdb1a8cb0c45d68c31 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:45.780745 1155 consensus_queue.cc:260] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:45.967080 773 raft_consensus.cc:2466] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:51:45.967336 1155 ts_tablet_manager.cc:1428] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.203s user 0.001s sys 0.000s
I20250629 01:51:45.967607 1155 tablet_bootstrap.cc:492] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:45.968724 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7e2c70221e94923b6a324a03f7133c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.968859 1158 ts_tablet_manager.cc:1918] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:45.968973 1158 log.cc:1199] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/7748846a3df14ffa801321ccde86e35f
I20250629 01:51:45.969280 1158 ts_tablet_manager.cc:1939] T 7748846a3df14ffa801321ccde86e35f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:45.970206 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 7748846a3df14ffa801321ccde86e35f (table default.Table_9edf5b31c7b24325930accee8cb627c3 [id=79a016a743f641158a9240b362c274c3]) successfully deleted
W20250629 01:51:45.970911 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet fbcae02d893f4795a12d26af57b18b85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcae02d893f4795a12d26af57b18b85 already in progress: creating tablet
W20250629 01:51:45.971375 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 683c87ea2ad44cecb2366d4a846d9195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 683c87ea2ad44cecb2366d4a846d9195 already in progress: creating tablet
I20250629 01:51:45.974090 1154 tablet_bootstrap.cc:654] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.978835 1598 sys_catalog.cc:455] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b76160e22c46929936cb9682322fae. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:45.979161 1598 sys_catalog.cc:458] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:45.983024 1466 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" instance_seqno: 1751161897763572) as {username='slave'} at 127.0.104.1:33307; Asking this server to re-register.
I20250629 01:51:45.985426 822 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:45.985453 1154 tablet_bootstrap.cc:492] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:45.985572 1154 ts_tablet_manager.cc:1397] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.001s
I20250629 01:51:45.987118 1547 leader_election.cc:290] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:45.987120 1154 raft_consensus.cc:357] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.987293 1154 raft_consensus.cc:383] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.987377 1154 raft_consensus.cc:738] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.987622 1154 consensus_queue.cc:260] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.987845 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a8a9bc691e4b158276f34a51c0dc61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:45.988063 1154 ts_tablet_manager.cc:1428] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:45.989022 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5e9f6de505a34afbae48a373b4b2a0cf (table default.Table_bd59d9e2715745828993ae11d0dd56d0 [id=9e7864d3513b4aca8f2b2b2cf84805bf]) successfully deleted
I20250629 01:51:45.991930 1466 ts_manager.cc:194] Registered new tserver with Master: ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:45.993009 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 49333302a27f4ac3b76a1e492b3c1b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49333302a27f4ac3b76a1e492b3c1b60 already in progress: creating tablet
I20250629 01:51:45.993664 1154 tablet_bootstrap.cc:492] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:45.994932 1159 tablet_bootstrap.cc:492] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:45.995043 1154 tablet_bootstrap.cc:654] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:45.995065 1159 ts_tablet_manager.cc:1397] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.285s user 0.025s sys 0.035s
I20250629 01:51:45.995729 1159 raft_consensus.cc:357] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.995872 1159 raft_consensus.cc:383] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:45.995944 1159 raft_consensus.cc:738] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:45.996155 1159 consensus_queue.cc:260] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:45.996623 1159 ts_tablet_manager.cc:1428] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:45.996868 1159 tablet_bootstrap.cc:492] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:45.999467 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8295e8178eff4d5eb95ac885092731e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8295e8178eff4d5eb95ac885092731e0 already in progress: creating tablet
I20250629 01:51:45.999886 1155 tablet_bootstrap.cc:654] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:46.000367 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet df66984acfa44a99bf69096aff6d49ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df66984acfa44a99bf69096aff6d49ab already in progress: creating tablet
W20250629 01:51:46.001185 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 683c87ea2ad44cecb2366d4a846d9195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 683c87ea2ad44cecb2366d4a846d9195 already in progress: creating tablet
W20250629 01:51:46.001906 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet fbcae02d893f4795a12d26af57b18b85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcae02d893f4795a12d26af57b18b85 already in progress: creating tablet
I20250629 01:51:46.008713 1154 tablet_bootstrap.cc:492] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.018972 1157 ts_tablet_manager.cc:1918] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.019096 1157 log.cc:1199] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a1938e67acaf4ab19fcbed13e1f1bfc6
I20250629 01:51:46.019757 1157 ts_tablet_manager.cc:1939] T a1938e67acaf4ab19fcbed13e1f1bfc6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:46.021202 1154 ts_tablet_manager.cc:1397] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250629 01:51:46.022709 1154 raft_consensus.cc:357] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.022938 1154 raft_consensus.cc:383] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.023021 1154 raft_consensus.cc:738] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.023259 1154 consensus_queue.cc:260] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.026324 1547 raft_consensus.cc:491] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:46.026638 1155 tablet_bootstrap.cc:492] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.026599 1547 raft_consensus.cc:513] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.026886 1155 ts_tablet_manager.cc:1397] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.004s
I20250629 01:51:46.027662 1155 raft_consensus.cc:357] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.027906 1155 raft_consensus.cc:383] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.027998 1155 raft_consensus.cc:738] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.028283 1155 consensus_queue.cc:260] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.028496 1159 tablet_bootstrap.cc:654] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.028856 1155 ts_tablet_manager.cc:1428] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:46.029486 1155 tablet_bootstrap.cc:492] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.032509 1155 tablet_bootstrap.cc:654] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.044243 1547 leader_election.cc:290] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:46.044363 1155 tablet_bootstrap.cc:492] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.044541 1155 ts_tablet_manager.cc:1397] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250629 01:51:46.045331 1640 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:46.046069 1640 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:51:46.047355 1155 raft_consensus.cc:357] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.047632 1154 ts_tablet_manager.cc:1428] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
W20250629 01:51:46.049255 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ea681bd91e54e4898bcfb72bed6198f consensus state has no leader
I20250629 01:51:46.051954 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 12b85f220f3947ca8110f2db937e7b68 (DEFAULT_TABLE table=default.Table_54de7744e36d4183a8af975cdc699bd3 [id=194bdf24776b480895832c6d00becb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.052340 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b85f220f3947ca8110f2db937e7b68. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.052403 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 consensus state has no leader
I20250629 01:51:46.051276 747 tablet_service.cc:1468] Processing CreateTablet for tablet 12b85f220f3947ca8110f2db937e7b68 (DEFAULT_TABLE table=default.Table_54de7744e36d4183a8af975cdc699bd3 [id=194bdf24776b480895832c6d00becb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.053784 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b85f220f3947ca8110f2db937e7b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.055936 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a75d7986152f427d88ed34a443128bc6 (DEFAULT_TABLE table=default.Table_0ed0327b20c3440b9f184793c8e1ee7d [id=c5569004f0ea4c808d805df1734decfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.051563 740 tablet_service.cc:1468] Processing CreateTablet for tablet a75d7986152f427d88ed34a443128bc6 (DEFAULT_TABLE table=default.Table_0ed0327b20c3440b9f184793c8e1ee7d [id=c5569004f0ea4c808d805df1734decfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.056245 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a75d7986152f427d88ed34a443128bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.056330 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a75d7986152f427d88ed34a443128bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.060688 739 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a (DEFAULT_TABLE table=default.Table_96a279d58b0a48b19a894d999572a340 [id=2ca8efc02f174cd8a1a375491a3a48ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.060995 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec92eddfabf40918a6cabaff5fe113a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.061201 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a (DEFAULT_TABLE table=default.Table_96a279d58b0a48b19a894d999572a340 [id=2ca8efc02f174cd8a1a375491a3a48ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.061420 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec92eddfabf40918a6cabaff5fe113a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.064316 755 tablet_service.cc:1468] Processing CreateTablet for tablet d6b6d5ff92c343cb8f1b3039b84390af (DEFAULT_TABLE table=default.Table_40c6d676eac2406887e3b761d5357587 [id=8038bbd131674ed59dbe36c73c14d466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.064620 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b6d5ff92c343cb8f1b3039b84390af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.064605 1019 tablet_service.cc:1468] Processing CreateTablet for tablet d6b6d5ff92c343cb8f1b3039b84390af (DEFAULT_TABLE table=default.Table_40c6d676eac2406887e3b761d5357587 [id=8038bbd131674ed59dbe36c73c14d466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.064884 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b6d5ff92c343cb8f1b3039b84390af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.069825 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 419) took 1074 ms. Trace:
I20250629 01:51:46.069945 1157 rpcz_store.cc:276] 0629 01:51:44.995806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:44.995859 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:45.793031 (+797172us) ts_tablet_manager.cc:1145] Deleting tablet a1938e67acaf4ab19fcbed13e1f1bfc6
0629 01:51:45.793037 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:45.933372 (+140335us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.018314 (+ 84942us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.069790 (+ 51476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":788819}
I20250629 01:51:46.069994 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eacbe3d626eb4bb980d04c2c6647aec9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.070214 1157 tablet_replica.cc:331] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:46.070472 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eacbe3d626eb4bb980d04c2c6647aec9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.070878 1157 raft_consensus.cc:2241] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.071030 1157 raft_consensus.cc:2270] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.071496 1157 ts_tablet_manager.cc:1905] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:46.072180 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddc24d0e668c49ac825b2578849ab2a3 consensus state has no leader
I20250629 01:51:46.072569 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_01a07db63e0242d6b1aba112aef1375d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.073104 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01a07db63e0242d6b1aba112aef1375d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.074424 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9a4eca53d6d5404fa56df4720474d29c" } modify_external_catalogs: true
I20250629 01:51:46.074479 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a9fddce63afb42b4be73a8c4807e050a" } modify_external_catalogs: true
I20250629 01:51:46.074569 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9a4eca53d6d5404fa56df4720474d29c" } modify_external_catalogs: true
I20250629 01:51:46.074658 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a9fddce63afb42b4be73a8c4807e050a" } modify_external_catalogs: true
I20250629 01:51:46.075819 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3f6fd2631c6241c6b6e205ccf4a313df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.076428 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:46.076445 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f6fd2631c6241c6b6e205ccf4a313df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.076720 1158 tablet_replica.cc:331] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.076884 1158 raft_consensus.cc:2241] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.077157 1158 raft_consensus.cc:2270] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.078130 1158 ts_tablet_manager.cc:1905] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.081873 1157 ts_tablet_manager.cc:1918] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.082007 1157 log.cc:1199] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/97439c6a6e844610b55b899966ae5b67
I20250629 01:51:46.082334 1157 ts_tablet_manager.cc:1939] T 97439c6a6e844610b55b899966ae5b67 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:46.082880 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 144173us
I20250629 01:51:46.083433 1155 raft_consensus.cc:383] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.083681 1155 raft_consensus.cc:738] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.084059 1155 consensus_queue.cc:260] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.084967 1155 ts_tablet_manager.cc:1428] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.040s user 0.001s sys 0.000s
I20250629 01:51:46.085709 1155 tablet_bootstrap.cc:492] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.086020 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e508393b9cb044c4b90194c8aab3184a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:46.085852 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d7b8cf118eb4a5c997815b97b9b7d00" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:46.086337 775 raft_consensus.cc:3058] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:46.087455 1155 tablet_bootstrap.cc:654] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.088779 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a1938e67acaf4ab19fcbed13e1f1bfc6 (table default.Table_38d08e8c1c0d4beea36513f66e9cbe43 [id=e12703cc30a94230b4c5f0fc21988359]) successfully deleted
I20250629 01:51:46.088871 775 raft_consensus.cc:2466] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:46.089462 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d7b8cf118eb4a5c997815b97b9b7d00" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:46.093478 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 425) took 1090 ms. Trace:
I20250629 01:51:46.093640 1157 rpcz_store.cc:276] 0629 01:51:45.002197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.002884 (+ 687us) service_pool.cc:225] Handling call
0629 01:51:46.070167 (+1067283us) ts_tablet_manager.cc:1145] Deleting tablet 97439c6a6e844610b55b899966ae5b67
0629 01:51:46.070175 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.074517 (+ 4342us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.081787 (+ 7270us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.093111 (+ 11324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052821}
I20250629 01:51:46.094257 896 tablet_service.cc:1468] Processing CreateTablet for tablet 12b85f220f3947ca8110f2db937e7b68 (DEFAULT_TABLE table=default.Table_54de7744e36d4183a8af975cdc699bd3 [id=194bdf24776b480895832c6d00becb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.094434 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 429) took 1091 ms. Trace:
I20250629 01:51:46.094645 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12b85f220f3947ca8110f2db937e7b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.095602 1158 ts_tablet_manager.cc:1918] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.095718 1158 log.cc:1199] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/6128463cd85d4f0cbfe21b0e5bd0e470
I20250629 01:51:46.095618 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e508393b9cb044c4b90194c8aab3184a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:46.096694 1048 raft_consensus.cc:3058] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:46.098556 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6bf3c7d0bb548e0b77ba8b569a72d0d consensus state has no leader
I20250629 01:51:46.094591 1157 rpcz_store.cc:276] 0629 01:51:45.002556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.003409 (+ 853us) service_pool.cc:225] Handling call
0629 01:51:46.094020 (+1090611us) ts_tablet_manager.cc:1145] Deleting tablet 8dd084685cc0407da9e12b0a4df386e9
0629 01:51:46.094028 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.094373 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075930}
I20250629 01:51:46.099756 1048 raft_consensus.cc:2466] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:51:46.099951 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 431) took 1024 ms. Trace:
I20250629 01:51:46.100059 1157 rpcz_store.cc:276] 0629 01:51:45.075758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.084382 (+ 8624us) service_pool.cc:225] Handling call
0629 01:51:46.099815 (+1015433us) ts_tablet_manager.cc:1145] Deleting tablet 45ca747f03524076b4112e4db78614fe
0629 01:51:46.099822 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.099900 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008800}
I20250629 01:51:46.100368 1157 tablet_replica.cc:331] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:46.100548 1157 raft_consensus.cc:2241] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.100731 1157 raft_consensus.cc:2270] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.101246 1157 ts_tablet_manager.cc:1905] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.102337 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
I20250629 01:51:46.102996 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a00a094ccfcf45879959855165452687" } new_table_name: "default.Table_20d462c5b8284f99b445c7539d756d3b" modify_external_catalogs: true
I20250629 01:51:46.106074 882 tablet_service.cc:1468] Processing CreateTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a (DEFAULT_TABLE table=default.Table_96a279d58b0a48b19a894d999572a340 [id=2ca8efc02f174cd8a1a375491a3a48ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.106463 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ec92eddfabf40918a6cabaff5fe113a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.106824 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 829c64e2c71347f4af2c8870cdd8af31 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.107075 1156 tablet_replica.cc:331] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.107266 1156 raft_consensus.cc:2241] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.107406 1156 raft_consensus.cc:2270] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.108160 1156 ts_tablet_manager.cc:1905] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.109570 1364 raft_consensus.cc:491] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:46.109683 1364 raft_consensus.cc:513] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.110358 1364 leader_election.cc:290] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:46.111416 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe14d1d54db74a99b05cf8b1a60827a9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:46.111996 986 leader_election.cc:343] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:46.112298 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:46.112288 850 leader_election.cc:343] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:46.112447 850 leader_election.cc:304] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:46.115264 1154 tablet_bootstrap.cc:492] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.115254 1156 ts_tablet_manager.cc:1918] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.115471 1156 log.cc:1199] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/829c64e2c71347f4af2c8870cdd8af31
I20250629 01:51:46.115783 1156 ts_tablet_manager.cc:1939] T 829c64e2c71347f4af2c8870cdd8af31 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.116592 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 829c64e2c71347f4af2c8870cdd8af31 (table default.Table_358f4795c7e244f2b72efe50dacd310f [id=00b7c47fc4684174b514feaab3b7d411]) successfully deleted
I20250629 01:51:46.119323 1158 ts_tablet_manager.cc:1939] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.120167 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 6128463cd85d4f0cbfe21b0e5bd0e470 (table default.Table_534d46b1a5ee4435844d4dd7c6626d1f [id=ff73871a84f745e383b03505bf30192d]) successfully deleted
I20250629 01:51:46.120504 1548 raft_consensus.cc:2802] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:46.120610 1548 raft_consensus.cc:695] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Becoming Leader. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Running, Role: LEADER
I20250629 01:51:46.120817 1548 consensus_queue.cc:237] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:46.122262 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8dd084685cc0407da9e12b0a4df386e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8dd084685cc0407da9e12b0a4df386e9 already in progress: creating tablet
W20250629 01:51:46.122723 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 45ca747f03524076b4112e4db78614fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45ca747f03524076b4112e4db78614fe already in progress: creating tablet
I20250629 01:51:46.124212 1154 tablet_bootstrap.cc:654] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.124290 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe14d1d54db74a99b05cf8b1a60827a9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:46.124485 916 raft_consensus.cc:2466] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:46.124887 1158 tablet_replica.cc:331] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:46.124977 850 leader_election.cc:343] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:46.125051 1158 raft_consensus.cc:2241] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.125156 850 leader_election.cc:304] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:46.125181 1158 raft_consensus.cc:2270] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.125409 1638 raft_consensus.cc:2747] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:46.125451 1155 tablet_bootstrap.cc:492] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.112764 882 tablet_service.cc:1468] Processing CreateTablet for tablet d6b6d5ff92c343cb8f1b3039b84390af (DEFAULT_TABLE table=default.Table_40c6d676eac2406887e3b761d5357587 [id=8038bbd131674ed59dbe36c73c14d466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.125982 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b6d5ff92c343cb8f1b3039b84390af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.125574 1155 ts_tablet_manager.cc:1397] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20250629 01:51:46.127167 1158 ts_tablet_manager.cc:1905] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.127214 1155 raft_consensus.cc:357] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.127892 1155 raft_consensus.cc:383] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.128079 1155 raft_consensus.cc:738] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.128486 1155 consensus_queue.cc:260] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.129362 1155 ts_tablet_manager.cc:1428] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:46.130040 1155 tablet_bootstrap.cc:492] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.132766 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2f7c1f1af39d47c191d34dc026f7f687
I20250629 01:51:46.133270 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a90b206d58d04acdb1a8cb0c45d68c31
I20250629 01:51:46.133648 447 catalog_manager.cc:6652] ReplaceTablet: tablet a90b206d58d04acdb1a8cb0c45d68c31 deleted and replaced by tablet bccdcd9de08648bb925db6fe56f97d20
I20250629 01:51:46.135180 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f6d79ac3c5f04b5db295c93c3aae92cb
I20250629 01:51:46.137738 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7cc18d1154ef4d05b3b82180a9c1ca8b
I20250629 01:51:46.138345 449 master_service.cc:948] ReplaceTablet: received request to replace tablet e962a020664b42ea918e5952dc43aade from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:46.139504 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 with delete_type TABLET_DATA_DELETED (replaced by tablet bccdcd9de08648bb925db6fe56f97d20) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.139667 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.139688 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.139871 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.140149 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 with delete_type TABLET_DATA_DELETED (replaced by tablet bccdcd9de08648bb925db6fe56f97d20) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.140161 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2845d8fbf3f34c769027b9b6bb3e1f08
I20250629 01:51:46.140463 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.140666 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 9fe9674ffeb44ba6ab60de59d4c08670 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:46.141343 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f6d79ac3c5f04b5db295c93c3aae92cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6d79ac3c5f04b5db295c93c3aae92cb already in progress: creating tablet
I20250629 01:51:46.141551 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:46.141563 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cc18d1154ef4d05b3b82180a9c1ca8b already in progress: creating tablet
I20250629 01:51:46.114499 896 tablet_service.cc:1468] Processing CreateTablet for tablet a75d7986152f427d88ed34a443128bc6 (DEFAULT_TABLE table=default.Table_0ed0327b20c3440b9f184793c8e1ee7d [id=c5569004f0ea4c808d805df1734decfc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:46.141746 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a90b206d58d04acdb1a8cb0c45d68c31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a90b206d58d04acdb1a8cb0c45d68c31 already in progress: creating tablet
I20250629 01:51:46.142354 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.142592 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.142803 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.143450 1155 tablet_bootstrap.cc:654] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:46.148170 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2f7c1f1af39d47c191d34dc026f7f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7c1f1af39d47c191d34dc026f7f687 already in progress: creating tablet
I20250629 01:51:46.151299 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b97cdfd111da4222af07e939cf5eae47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.151486 988 leader_election.cc:304] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:46.151672 988 raft_consensus.cc:2684] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:46.153533 1159 tablet_bootstrap.cc:492] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:46.153813 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a75d7986152f427d88ed34a443128bc6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.153935 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2845d8fbf3f34c769027b9b6bb3e1f08 already in progress: creating tablet
I20250629 01:51:46.154081 1159 ts_tablet_manager.cc:1397] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.001s
W20250629 01:51:46.155597 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bbbeb43473a4f6f892056a30f62589f consensus state has no leader
I20250629 01:51:46.157184 1159 raft_consensus.cc:357] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.157423 1159 raft_consensus.cc:383] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.157646 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_62002e6a41684d369e22fb34ba498d8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.158187 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b97cdfd111da4222af07e939cf5eae47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.157552 1159 raft_consensus.cc:738] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.159150 1159 consensus_queue.cc:260] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.159750 1155 tablet_bootstrap.cc:492] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.159936 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 97439c6a6e844610b55b899966ae5b67 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b0782c7e652419096710563930f2072) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:46.159984 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62002e6a41684d369e22fb34ba498d8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.160161 1156 tablet_replica.cc:331] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.160321 1156 raft_consensus.cc:2241] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.160465 1156 raft_consensus.cc:2270] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.160523 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af7702628dd2458d831dcc0ce1be9327"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.161373 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af7702628dd2458d831dcc0ce1be9327 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.161640 1156 ts_tablet_manager.cc:1905] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.161670 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.162010 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 with delete_type TABLET_DATA_DELETED (replaced by tablet bccdcd9de08648bb925db6fe56f97d20) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.162289 883 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.162549 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.162793 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.159999 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0ad69c6d3ce44bfd9da5075795ba23a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.163121 1638 raft_consensus.cc:491] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:46.163247 1638 raft_consensus.cc:513] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.163753 1638 leader_election.cc:290] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:46.165903 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "299922fa5e694bdf96f29f1522068656" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:46.166527 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "299922fa5e694bdf96f29f1522068656" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:46.166989 850 leader_election.cc:343] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:46.167160 850 leader_election.cc:343] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:46.167263 850 leader_election.cc:304] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:46.167778 1154 tablet_bootstrap.cc:492] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.167902 1154 ts_tablet_manager.cc:1397] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20250629 01:51:46.168581 1154 raft_consensus.cc:357] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:46.168648 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ad69c6d3ce44bfd9da5075795ba23a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.168735 1154 raft_consensus.cc:383] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.168819 1154 raft_consensus.cc:738] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.169019 1154 consensus_queue.cc:260] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.169835 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 7748846a3df14ffa801321ccde86e35f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:46.169862 1638 raft_consensus.cc:2747] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:46.169989 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9200a719f8584b2899fbb70d1be7843c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.171900 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9200a719f8584b2899fbb70d1be7843c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.172933 739 tablet_service.cc:1515] Processing DeleteTablet for tablet a1938e67acaf4ab19fcbed13e1f1bfc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.173040 1154 ts_tablet_manager.cc:1428] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:46.176955 1156 ts_tablet_manager.cc:1918] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.177323 1156 log.cc:1199] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/97439c6a6e844610b55b899966ae5b67
I20250629 01:51:46.177852 1156 ts_tablet_manager.cc:1939] T 97439c6a6e844610b55b899966ae5b67 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.179364 454 master_service.cc:948] ReplaceTablet: received request to replace tablet e3b94f8250a143b096703f2b0386e3cd from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:46.178915 1155 ts_tablet_manager.cc:1397] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.003s
I20250629 01:51:46.179648 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 97439c6a6e844610b55b899966ae5b67 with delete_type TABLET_DATA_DELETED (replaced by tablet 1b0782c7e652419096710563930f2072) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.179996 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 97439c6a6e844610b55b899966ae5b67 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
I20250629 01:51:46.180424 1154 tablet_bootstrap.cc:492] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.180617 1156 tablet_replica.cc:331] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.181196 1156 raft_consensus.cc:2241] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.181676 1159 ts_tablet_manager.cc:1428] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250629 01:51:46.183528 1154 tablet_bootstrap.cc:654] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.183640 1159 tablet_bootstrap.cc:492] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:46.184609 1158 ts_tablet_manager.cc:1918] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.184721 1158 log.cc:1199] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/fe14d1d54db74a99b05cf8b1a60827a9
I20250629 01:51:46.185022 1158 ts_tablet_manager.cc:1939] T fe14d1d54db74a99b05cf8b1a60827a9 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.186455 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet fe14d1d54db74a99b05cf8b1a60827a9 (table default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]) successfully deleted
W20250629 01:51:46.186664 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a90b206d58d04acdb1a8cb0c45d68c31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a90b206d58d04acdb1a8cb0c45d68c31 already in progress: creating tablet
W20250629 01:51:46.186882 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2f7c1f1af39d47c191d34dc026f7f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7c1f1af39d47c191d34dc026f7f687 already in progress: creating tablet
W20250629 01:51:46.187057 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f6d79ac3c5f04b5db295c93c3aae92cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6d79ac3c5f04b5db295c93c3aae92cb already in progress: creating tablet
I20250629 01:51:46.187706 1156 raft_consensus.cc:2270] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:46.187788 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2845d8fbf3f34c769027b9b6bb3e1f08 already in progress: creating tablet
I20250629 01:51:46.188464 1159 tablet_bootstrap.cc:654] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.191776 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
I20250629 01:51:46.192047 449 catalog_manager.cc:5582] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 reported cstate change: term changed from 0 to 1, leader changed from <none> to 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:46.197292 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b1ebbcb2c26b4667b2405154a9d8fccb (DEFAULT_TABLE table=default.Table_7f38778518cf47bca0a3dc0dc8e94313 [id=70f6cbc3d10848f9af1a4c482c27eeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.197355 739 tablet_service.cc:1468] Processing CreateTablet for tablet b1ebbcb2c26b4667b2405154a9d8fccb (DEFAULT_TABLE table=default.Table_7f38778518cf47bca0a3dc0dc8e94313 [id=70f6cbc3d10848f9af1a4c482c27eeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.197683 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ebbcb2c26b4667b2405154a9d8fccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.197623 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ef3060d46ac24330be49103920dc1dee (DEFAULT_TABLE table=default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.197826 1156 ts_tablet_manager.cc:1905] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.197623 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ebbcb2c26b4667b2405154a9d8fccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.187515 1155 raft_consensus.cc:357] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.200567 1155 raft_consensus.cc:383] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.200692 740 tablet_service.cc:1468] Processing CreateTablet for tablet ef3060d46ac24330be49103920dc1dee (DEFAULT_TABLE table=default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.200711 1155 raft_consensus.cc:738] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.201159 1155 consensus_queue.cc:260] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.202617 1155 ts_tablet_manager.cc:1428] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.023s user 0.000s sys 0.001s
I20250629 01:51:46.202858 1155 tablet_bootstrap.cc:492] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.204203 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 6128463cd85d4f0cbfe21b0e5bd0e470 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.205616 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3060d46ac24330be49103920dc1dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.207139 1156 ts_tablet_manager.cc:1918] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.204210 1155 tablet_bootstrap.cc:654] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.208515 1156 log.cc:1199] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a1938e67acaf4ab19fcbed13e1f1bfc6
I20250629 01:51:46.208854 1156 ts_tablet_manager.cc:1939] T a1938e67acaf4ab19fcbed13e1f1bfc6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.209648 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a1938e67acaf4ab19fcbed13e1f1bfc6 (table default.Table_38d08e8c1c0d4beea36513f66e9cbe43 [id=e12703cc30a94230b4c5f0fc21988359]) successfully deleted
I20250629 01:51:46.210516 1156 tablet_replica.cc:331] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.210887 1156 raft_consensus.cc:2241] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.211277 1156 raft_consensus.cc:2270] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.212206 1156 ts_tablet_manager.cc:1905] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.216465 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3060d46ac24330be49103920dc1dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.217296 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9fe9674ffeb44ba6ab60de59d4c08670
I20250629 01:51:46.217676 446 catalog_manager.cc:6652] ReplaceTablet: tablet 9fe9674ffeb44ba6ab60de59d4c08670 deleted and replaced by tablet f465d0bb46674763abfc8d93f45cd13d
I20250629 01:51:46.217960 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 07e0727102d54294a7c13a4aeef311e1
I20250629 01:51:46.219324 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.219542 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e0727102d54294a7c13a4aeef311e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.219942 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.220077 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e0727102d54294a7c13a4aeef311e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.223779 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e3b94f8250a143b096703f2b0386e3cd
I20250629 01:51:46.224159 454 catalog_manager.cc:6652] ReplaceTablet: tablet e3b94f8250a143b096703f2b0386e3cd deleted and replaced by tablet c5771fdd618e45118df2b6dab7f668d9
I20250629 01:51:46.224601 739 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b94f8250a143b096703f2b0386e3cd with delete_type TABLET_DATA_DELETED (replaced by tablet c5771fdd618e45118df2b6dab7f668d9) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.224752 1157 ts_tablet_manager.cc:1918] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.224870 1157 log.cc:1199] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/829c64e2c71347f4af2c8870cdd8af31
I20250629 01:51:46.225203 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b94f8250a143b096703f2b0386e3cd with delete_type TABLET_DATA_DELETED (replaced by tablet c5771fdd618e45118df2b6dab7f668d9) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.226243 1158 tablet_replica.cc:331] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:46.226298 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cc18d1154ef4d05b3b82180a9c1ca8b already in progress: creating tablet
I20250629 01:51:46.226392 1158 raft_consensus.cc:2241] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.226549 1158 raft_consensus.cc:2270] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.226578 1157 ts_tablet_manager.cc:1939] T 829c64e2c71347f4af2c8870cdd8af31 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:46.227044 1638 raft_consensus.cc:491] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:46.227761 1638 raft_consensus.cc:513] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.228602 1638 leader_election.cc:290] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:46.228768 1158 ts_tablet_manager.cc:1905] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.231032 882 tablet_service.cc:1468] Processing CreateTablet for tablet b1ebbcb2c26b4667b2405154a9d8fccb (DEFAULT_TABLE table=default.Table_7f38778518cf47bca0a3dc0dc8e94313 [id=70f6cbc3d10848f9af1a4c482c27eeb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.232913 1155 tablet_bootstrap.cc:492] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.233132 1155 ts_tablet_manager.cc:1397] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.004s
I20250629 01:51:46.233898 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 432) took 1099 ms. Trace:
I20250629 01:51:46.233919 1155 raft_consensus.cc:357] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.234558 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ebbcb2c26b4667b2405154a9d8fccb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.234572 1155 raft_consensus.cc:383] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.234715 1155 raft_consensus.cc:738] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.234977 1155 consensus_queue.cc:260] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.235715 1156 ts_tablet_manager.cc:1918] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.235858 1156 log.cc:1199] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/6128463cd85d4f0cbfe21b0e5bd0e470
I20250629 01:51:46.236229 1156 ts_tablet_manager.cc:1939] T 6128463cd85d4f0cbfe21b0e5bd0e470 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.238148 1157 rpcz_store.cc:276] 0629 01:51:45.127803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.135938 (+ 8135us) service_pool.cc:225] Handling call
0629 01:51:46.100309 (+964371us) ts_tablet_manager.cc:1145] Deleting tablet 829c64e2c71347f4af2c8870cdd8af31
0629 01:51:46.100316 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.104535 (+ 4219us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.207429 (+102894us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.227682 (+ 20253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":964147}
I20250629 01:51:46.238421 1154 tablet_bootstrap.cc:492] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.238476 1157 tablet_replica.cc:331] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:46.238543 1154 ts_tablet_manager.cc:1397] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20250629 01:51:46.238778 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 6128463cd85d4f0cbfe21b0e5bd0e470 (table default.Table_534d46b1a5ee4435844d4dd7c6626d1f [id=ff73871a84f745e383b03505bf30192d]) successfully deleted
I20250629 01:51:46.239163 1154 raft_consensus.cc:357] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.239341 1154 raft_consensus.cc:383] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.239405 1154 raft_consensus.cc:738] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.239864 1157 raft_consensus.cc:2241] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:46.239918 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9fe9674ffeb44ba6ab60de59d4c08670 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fe9674ffeb44ba6ab60de59d4c08670 already in progress: creating tablet
I20250629 01:51:46.240002 1157 raft_consensus.cc:2270] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.240235 1154 consensus_queue.cc:260] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:46.240473 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 07e0727102d54294a7c13a4aeef311e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07e0727102d54294a7c13a4aeef311e1 already in progress: creating tablet
I20250629 01:51:46.240538 1157 ts_tablet_manager.cc:1905] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:46.240931 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e3b94f8250a143b096703f2b0386e3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b94f8250a143b096703f2b0386e3cd already in progress: creating tablet
I20250629 01:51:46.245177 1154 ts_tablet_manager.cc:1428] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250629 01:51:46.245613 1158 ts_tablet_manager.cc:1918] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.245733 1158 log.cc:1199] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/97439c6a6e844610b55b899966ae5b67
I20250629 01:51:46.246058 1158 ts_tablet_manager.cc:1939] T 97439c6a6e844610b55b899966ae5b67 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.248695 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ec3937679b37476f934aceb30ac62d68 (DEFAULT_TABLE table=default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.249011 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3937679b37476f934aceb30ac62d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.249236 1154 tablet_bootstrap.cc:492] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.250806 1154 tablet_bootstrap.cc:654] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.251638 747 tablet_service.cc:1468] Processing CreateTablet for tablet ec3937679b37476f934aceb30ac62d68 (DEFAULT_TABLE table=default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.251977 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3937679b37476f934aceb30ac62d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.252848 739 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.253360 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 97439c6a6e844610b55b899966ae5b67 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
W20250629 01:51:46.253587 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9fe9674ffeb44ba6ab60de59d4c08670 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9fe9674ffeb44ba6ab60de59d4c08670 already in progress: creating tablet
W20250629 01:51:46.253785 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 07e0727102d54294a7c13a4aeef311e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07e0727102d54294a7c13a4aeef311e1 already in progress: creating tablet
W20250629 01:51:46.253989 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e3b94f8250a143b096703f2b0386e3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b94f8250a143b096703f2b0386e3cd already in progress: creating tablet
I20250629 01:51:46.254117 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.254534 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 with delete_type TABLET_DATA_DELETED (replaced by tablet e82b1352bb4d49769808dc402c85890f) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.255319 1157 ts_tablet_manager.cc:1918] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.255420 1157 log.cc:1199] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/7748846a3df14ffa801321ccde86e35f
I20250629 01:51:46.255698 1157 ts_tablet_manager.cc:1939] T 7748846a3df14ffa801321ccde86e35f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:46.255965 1158 tablet_replica.cc:331] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.256143 1158 raft_consensus.cc:2241] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.256258 1158 raft_consensus.cc:2270] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.256316 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 433) took 1082 ms. Trace:
I20250629 01:51:46.256392 1157 rpcz_store.cc:276] 0629 01:51:45.173823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.173883 (+ 60us) service_pool.cc:225] Handling call
0629 01:51:46.238425 (+1064542us) ts_tablet_manager.cc:1145] Deleting tablet 7748846a3df14ffa801321ccde86e35f
0629 01:51:46.238433 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.243587 (+ 5154us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.255239 (+ 11652us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.256271 (+ 1032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062762}
I20250629 01:51:46.256673 1157 tablet_replica.cc:331] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:46.256850 1157 raft_consensus.cc:2241] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.256995 1157 raft_consensus.cc:2270] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.257359 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e0727102d54294a7c13a4aeef311e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.257571 1021 tablet_service.cc:1468] Processing CreateTablet for tablet eb26725bd175429d805947c4eda6e40a (DEFAULT_TABLE table=default.Table_b97cdfd111da4222af07e939cf5eae47 [id=d692ec01b04c427095ce8968280485c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.257665 739 tablet_service.cc:1468] Processing CreateTablet for tablet eb26725bd175429d805947c4eda6e40a (DEFAULT_TABLE table=default.Table_b97cdfd111da4222af07e939cf5eae47 [id=d692ec01b04c427095ce8968280485c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.257869 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb26725bd175429d805947c4eda6e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.257943 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb26725bd175429d805947c4eda6e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.260006 1016 tablet_service.cc:1468] Processing CreateTablet for tablet bbe1f8083a884cdd97afce83bd22073c (DEFAULT_TABLE table=default.Table_3f6fd2631c6241c6b6e205ccf4a313df [id=6325ae6cb8a245df89d42a498b2a3828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.260313 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe1f8083a884cdd97afce83bd22073c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.260269 740 tablet_service.cc:1468] Processing CreateTablet for tablet bbe1f8083a884cdd97afce83bd22073c (DEFAULT_TABLE table=default.Table_3f6fd2631c6241c6b6e205ccf4a313df [id=6325ae6cb8a245df89d42a498b2a3828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.260547 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe1f8083a884cdd97afce83bd22073c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.261112 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f9e7762f7fba4fc59cd1dd954b009b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.261570 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9e7762f7fba4fc59cd1dd954b009b6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.264497 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_01a07db63e0242d6b1aba112aef1375d" } new_table_name: "default.Table_39a7d1cee49d4af2ab7db17efbf44268" modify_external_catalogs: true
W20250629 01:51:46.267557 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet fe14d1d54db74a99b05cf8b1a60827a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe14d1d54db74a99b05cf8b1a60827a9 already in progress: creating tablet
I20250629 01:51:46.268056 1154 tablet_bootstrap.cc:492] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.268213 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b94f8250a143b096703f2b0386e3cd with delete_type TABLET_DATA_DELETED (replaced by tablet c5771fdd618e45118df2b6dab7f668d9) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.268256 1154 ts_tablet_manager.cc:1397] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20250629 01:51:46.268711 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 (DEFAULT_TABLE table=default.Table_eacbe3d626eb4bb980d04c2c6647aec9 [id=22db5880e03948678f7bcf4aa8082579]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.269053 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e90a48af8f48bba3b3fd4f4110d367. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.269497 883 tablet_service.cc:1468] Processing CreateTablet for tablet ec3937679b37476f934aceb30ac62d68 (DEFAULT_TABLE table=default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.269809 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec3937679b37476f934aceb30ac62d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.275043 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_05483b691b564e7482b9d04d0afb1b29" } new_table_name: "default.Table_eab06a5f9cbc4248a1757e3b5518545b" modify_external_catalogs: true
I20250629 01:51:46.277117 1158 ts_tablet_manager.cc:1905] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.280155 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b1a50d6abfb64bf5a297be7fc5c93376" } new_table_name: "default.Table_1e578e5fe38c40d38ba9b7759798a267" modify_external_catalogs: true
I20250629 01:51:46.269225 1154 raft_consensus.cc:357] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.281003 883 tablet_service.cc:1468] Processing CreateTablet for tablet eb26725bd175429d805947c4eda6e40a (DEFAULT_TABLE table=default.Table_b97cdfd111da4222af07e939cf5eae47 [id=d692ec01b04c427095ce8968280485c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.281168 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9279e8fb53d14dfba52c6d7ae60a1ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.281309 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb26725bd175429d805947c4eda6e40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.280488 1154 raft_consensus.cc:383] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:51:46.281600 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9279e8fb53d14dfba52c6d7ae60a1ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.284147 1154 raft_consensus.cc:738] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.284366 1154 consensus_queue.cc:260] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:46.285214 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01d95356c7a7419bb56ddf67d451eadc consensus state has no leader
I20250629 01:51:46.285969 896 tablet_service.cc:1468] Processing CreateTablet for tablet bbe1f8083a884cdd97afce83bd22073c (DEFAULT_TABLE table=default.Table_3f6fd2631c6241c6b6e205ccf4a313df [id=6325ae6cb8a245df89d42a498b2a3828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.286321 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbe1f8083a884cdd97afce83bd22073c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.289913 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet dfee69f9484046d8abe10d2f07033e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.290410 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79e61d9576f8414ebf0074d57cdfb1f5" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:46.290452 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 01d95356c7a7419bb56ddf67d451eadc (DEFAULT_TABLE table=default.Table_39a7d1cee49d4af2ab7db17efbf44268 [id=3f6cd9a247f34a6d84c591bca6e7a4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.290707 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01d95356c7a7419bb56ddf67d451eadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.291100 1154 ts_tablet_manager.cc:1428] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250629 01:51:46.292200 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79e61d9576f8414ebf0074d57cdfb1f5" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:46.293519 882 tablet_service.cc:1468] Processing CreateTablet for tablet ef3060d46ac24330be49103920dc1dee (DEFAULT_TABLE table=default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.293848 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3060d46ac24330be49103920dc1dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.296314 1155 ts_tablet_manager.cc:1428] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20250629 01:51:46.296613 1155 tablet_bootstrap.cc:492] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.296828 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 829c64e2c71347f4af2c8870cdd8af31 (table default.Table_358f4795c7e244f2b72efe50dacd310f [id=00b7c47fc4684174b514feaab3b7d411]) successfully deleted
I20250629 01:51:46.297134 1154 tablet_bootstrap.cc:492] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.297175 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 7748846a3df14ffa801321ccde86e35f (table default.Table_9edf5b31c7b24325930accee8cb627c3 [id=79a016a743f641158a9240b362c274c3]) successfully deleted
I20250629 01:51:46.298617 1154 tablet_bootstrap.cc:654] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:46.299589 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4d6f1eb43b24fda94aa0afcf4501195 consensus state has no leader
I20250629 01:51:46.302613 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e3283d56ec2743e2a1073ee8909e103f" } new_table_name: "default.Table_2c0e663e50e24133b9794fc3ab9022c6" modify_external_catalogs: true
I20250629 01:51:46.304054 877 tablet_service.cc:1468] Processing CreateTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 (DEFAULT_TABLE table=default.Table_eacbe3d626eb4bb980d04c2c6647aec9 [id=22db5880e03948678f7bcf4aa8082579]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.304368 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e90a48af8f48bba3b3fd4f4110d367. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.290311 740 tablet_service.cc:1468] Processing CreateTablet for tablet 01d95356c7a7419bb56ddf67d451eadc (DEFAULT_TABLE table=default.Table_39a7d1cee49d4af2ab7db17efbf44268 [id=3f6cd9a247f34a6d84c591bca6e7a4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.307437 895 tablet_service.cc:1468] Processing CreateTablet for tablet 01d95356c7a7419bb56ddf67d451eadc (DEFAULT_TABLE table=default.Table_39a7d1cee49d4af2ab7db17efbf44268 [id=3f6cd9a247f34a6d84c591bca6e7a4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.307730 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01d95356c7a7419bb56ddf67d451eadc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.308274 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 8cced2e8c94345b88f388becb0060d7a (DEFAULT_TABLE table=default.Table_e43be787310e423097abae89ed26eb57 [id=fdc0be4274344566bcf92d324926d457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.308547 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cced2e8c94345b88f388becb0060d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.313040 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 (DEFAULT_TABLE table=default.Table_af7702628dd2458d831dcc0ce1be9327 [id=ae19e8ff3bdb4232969c0b35da790b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.313454 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7b76cb56b24604b13ee29c7f576ce2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.314075 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c6239dd09a343d59da51d114f48658a consensus state has no leader
I20250629 01:51:46.317441 1158 ts_tablet_manager.cc:1918] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.317663 1158 log.cc:1199] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/79e61d9576f8414ebf0074d57cdfb1f5
I20250629 01:51:46.318056 1158 ts_tablet_manager.cc:1939] T 79e61d9576f8414ebf0074d57cdfb1f5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.318356 1027 tablet_service.cc:1468] Processing CreateTablet for tablet bb98aa9d365541969ea1dc70960760b2 (DEFAULT_TABLE table=default.Table_62002e6a41684d369e22fb34ba498d8f [id=631e5b6c35a7419a8963c0a4e468e439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.318428 1016 tablet_service.cc:1468] Processing CreateTablet for tablet bccdcd9de08648bb925db6fe56f97d20 (DEFAULT_TABLE table=default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.318670 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb98aa9d365541969ea1dc70960760b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.318955 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 79e61d9576f8414ebf0074d57cdfb1f5 (table default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]) successfully deleted
W20250629 01:51:46.319680 850 leader_election.cc:343] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:46.319887 850 leader_election.cc:343] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:46.319964 850 leader_election.cc:304] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:46.321341 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 021da4395e414413b9b11ebcfbffcd19 (DEFAULT_TABLE table=default.Table_0ad69c6d3ce44bfd9da5075795ba23a5 [id=e22c0b2a20924851a62539cfb7d6303e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.321645 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021da4395e414413b9b11ebcfbffcd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.323637 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bccdcd9de08648bb925db6fe56f97d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.326115 1158 tablet_replica.cc:331] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.326335 1158 raft_consensus.cc:2241] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.326514 1158 raft_consensus.cc:2270] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.326987 1638 raft_consensus.cc:2747] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:46.327147 1158 ts_tablet_manager.cc:1905] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.328075 891 tablet_service.cc:1468] Processing CreateTablet for tablet 8cced2e8c94345b88f388becb0060d7a (DEFAULT_TABLE table=default.Table_e43be787310e423097abae89ed26eb57 [id=fdc0be4274344566bcf92d324926d457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.328382 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cced2e8c94345b88f388becb0060d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.332495 1158 ts_tablet_manager.cc:1918] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.332582 1158 log.cc:1199] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/dfee69f9484046d8abe10d2f07033e09
I20250629 01:51:46.332522 879 tablet_service.cc:1468] Processing CreateTablet for tablet 021da4395e414413b9b11ebcfbffcd19 (DEFAULT_TABLE table=default.Table_0ad69c6d3ce44bfd9da5075795ba23a5 [id=e22c0b2a20924851a62539cfb7d6303e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.332839 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021da4395e414413b9b11ebcfbffcd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.332902 1158 ts_tablet_manager.cc:1939] T dfee69f9484046d8abe10d2f07033e09 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.333660 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet dfee69f9484046d8abe10d2f07033e09 (table default.Table_2bd4de753ed04021b2e3122b263bafab [id=40b322522a664b8f8bb8f39388b8ddf6]) successfully deleted
I20250629 01:51:46.334738 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2ba661f263ce4cbda78e8be06f7d662d" } modify_external_catalogs: true
I20250629 01:51:46.334883 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2ba661f263ce4cbda78e8be06f7d662d" } modify_external_catalogs: true
I20250629 01:51:46.341993 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fed4bed522aa4c8192e79303877c5472" } modify_external_catalogs: true
I20250629 01:51:46.342154 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fed4bed522aa4c8192e79303877c5472" } modify_external_catalogs: true
I20250629 01:51:46.342577 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a968452543bd46baa06280c24819ada1" } modify_external_catalogs: true
I20250629 01:51:46.342711 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a968452543bd46baa06280c24819ada1" } modify_external_catalogs: true
I20250629 01:51:46.343911 1154 tablet_bootstrap.cc:492] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.344058 1154 ts_tablet_manager.cc:1397] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20250629 01:51:46.344753 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8a1e4394a537495b9bf3f9e4335f7936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.344996 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_97c5167bc43d47699812760d5d1939c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.345132 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_737eb1cf8b184a14b253c25d114a14b7" } modify_external_catalogs: true
I20250629 01:51:46.345260 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_737eb1cf8b184a14b253c25d114a14b7" } modify_external_catalogs: true
W20250629 01:51:46.345242 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a1e4394a537495b9bf3f9e4335f7936 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.345415 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c5167bc43d47699812760d5d1939c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.345506 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d11d1c20ae434e399df06213b6217417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.345952 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d11d1c20ae434e399df06213b6217417 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.346556 1154 raft_consensus.cc:357] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.346732 1154 raft_consensus.cc:383] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.346851 1154 raft_consensus.cc:738] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.347074 1154 consensus_queue.cc:260] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.347558 1154 ts_tablet_manager.cc:1428] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:46.349186 1154 tablet_bootstrap.cc:492] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.350518 1154 tablet_bootstrap.cc:654] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.352157 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 (DEFAULT_TABLE table=default.Table_af7702628dd2458d831dcc0ce1be9327 [id=ae19e8ff3bdb4232969c0b35da790b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.352471 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7b76cb56b24604b13ee29c7f576ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.356182 1157 ts_tablet_manager.cc:1905] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.358158 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9279e8fb53d14dfba52c6d7ae60a1ad5" } new_table_name: "default.Table_2d471323a04d4c0389ad1d253aa5cd36" modify_external_catalogs: true
I20250629 01:51:46.356778 890 tablet_service.cc:1468] Processing CreateTablet for tablet bccdcd9de08648bb925db6fe56f97d20 (DEFAULT_TABLE table=default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.358608 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bccdcd9de08648bb925db6fe56f97d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.357455 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.362756 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1c995ce14c9a4966b842fcba520cf87d
I20250629 01:51:46.371598 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c995ce14c9a4966b842fcba520cf87d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:46.374429 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ca02bcd9467496c9057b3de9c1b1725 has no consensus state
W20250629 01:51:46.374686 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1c995ce14c9a4966b842fcba520cf87d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c995ce14c9a4966b842fcba520cf87d already in progress: creating tablet
I20250629 01:51:46.376389 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_b39b700fff9245d8a067319ab01e5d8e [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 5 got 0
I20250629 01:51:46.380545 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 (DEFAULT_TABLE table=default.Table_f9e7762f7fba4fc59cd1dd954b009b6b [id=d342099d88ee4e3ebbc5d5a1f17ba4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.380867 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed1ee73bb8f4761806a66f988b57bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.383107 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d1a8a9bc691e4b158276f34a51c0dc61
I20250629 01:51:46.384459 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.384689 1154 tablet_bootstrap.cc:492] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.384799 1154 ts_tablet_manager.cc:1397] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20250629 01:51:46.385229 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01d95356c7a7419bb56ddf67d451eadc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.385239 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d1a8a9bc691e4b158276f34a51c0dc61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1a8a9bc691e4b158276f34a51c0dc61 already in progress: creating tablet
I20250629 01:51:46.385505 1154 raft_consensus.cc:357] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.385639 1154 raft_consensus.cc:383] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.385702 1154 raft_consensus.cc:738] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.385907 1154 consensus_queue.cc:260] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.386662 1154 ts_tablet_manager.cc:1428] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:46.387521 1154 tablet_bootstrap.cc:492] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.388876 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d7a4536b4ffa4887a2b30a00f22275c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.388981 747 tablet_service.cc:1468] Processing CreateTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 (DEFAULT_TABLE table=default.Table_eacbe3d626eb4bb980d04c2c6647aec9 [id=22db5880e03948678f7bcf4aa8082579]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.389068 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_82be402325034e178ca2423c9729a2dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.389317 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7a4536b4ffa4887a2b30a00f22275c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.389518 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82be402325034e178ca2423c9729a2dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.391168 744 tablet_service.cc:1468] Processing CreateTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 (DEFAULT_TABLE table=default.Table_af7702628dd2458d831dcc0ce1be9327 [id=ae19e8ff3bdb4232969c0b35da790b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.390929 749 tablet_service.cc:1468] Processing CreateTablet for tablet 021da4395e414413b9b11ebcfbffcd19 (DEFAULT_TABLE table=default.Table_0ad69c6d3ce44bfd9da5075795ba23a5 [id=e22c0b2a20924851a62539cfb7d6303e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.391645 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
I20250629 01:51:46.392024 755 tablet_service.cc:1515] Processing DeleteTablet for tablet fe14d1d54db74a99b05cf8b1a60827a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c995ce14c9a4966b842fcba520cf87d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.392679 1156 tablet_replica.cc:331] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.392693 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_df0fc31afd154b05a7d3d84bec25e140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.392621 743 tablet_service.cc:1468] Processing CreateTablet for tablet bccdcd9de08648bb925db6fe56f97d20 (DEFAULT_TABLE table=default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:46.393096 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df0fc31afd154b05a7d3d84bec25e140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.391270 751 tablet_service.cc:1468] Processing CreateTablet for tablet 8cced2e8c94345b88f388becb0060d7a (DEFAULT_TABLE table=default.Table_e43be787310e423097abae89ed26eb57 [id=fdc0be4274344566bcf92d324926d457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.393448 1156 raft_consensus.cc:2241] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.394034 1156 raft_consensus.cc:2270] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.394439 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e90a48af8f48bba3b3fd4f4110d367. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.394675 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.395064 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.395782 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cced2e8c94345b88f388becb0060d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.396206 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bccdcd9de08648bb925db6fe56f97d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.396256 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_bb5794f5710541f0adb81634de0ead93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.396595 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb5794f5710541f0adb81634de0ead93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.397015 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021da4395e414413b9b11ebcfbffcd19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.398103 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c7b76cb56b24604b13ee29c7f576ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.398697 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0
I20250629 01:51:46.399396 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.394675 745 tablet_service.cc:1468] Processing CreateTablet for tablet bb98aa9d365541969ea1dc70960760b2 (DEFAULT_TABLE table=default.Table_62002e6a41684d369e22fb34ba498d8f [id=631e5b6c35a7419a8963c0a4e468e439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.399955 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb98aa9d365541969ea1dc70960760b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.399925 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 already in progress: creating tablet
I20250629 01:51:46.402508 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ecbfc4e45dc9435bbd7a55ba7b7e4155
I20250629 01:51:46.404024 1154 tablet_bootstrap.cc:654] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.404341 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.405133 455 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
W20250629 01:51:46.405946 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 already in progress: creating tablet
I20250629 01:51:46.406118 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c995ce14c9a4966b842fcba520cf87d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.416517 1156 ts_tablet_manager.cc:1905] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.420624 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_96a279d58b0a48b19a894d999572a340" } modify_external_catalogs: true
I20250629 01:51:46.420789 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_96a279d58b0a48b19a894d999572a340" } modify_external_catalogs: true
I20250629 01:51:46.422284 1156 ts_tablet_manager.cc:1918] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.422399 1156 log.cc:1199] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/fe14d1d54db74a99b05cf8b1a60827a9
I20250629 01:51:46.422703 1156 ts_tablet_manager.cc:1939] T fe14d1d54db74a99b05cf8b1a60827a9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.429391 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2ec92eddfabf40918a6cabaff5fe113a
I20250629 01:51:46.430625 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:46.431157 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2ec92eddfabf40918a6cabaff5fe113a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ec92eddfabf40918a6cabaff5fe113a already in progress: creating tablet
I20250629 01:51:46.436795 1154 tablet_bootstrap.cc:492] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.437073 1154 ts_tablet_manager.cc:1397] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.001s
I20250629 01:51:46.437829 1154 raft_consensus.cc:357] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.438035 1154 raft_consensus.cc:383] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.438125 1154 raft_consensus.cc:738] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.438359 1154 consensus_queue.cc:260] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.438838 1154 ts_tablet_manager.cc:1428] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:46.439710 1016 tablet_service.cc:1468] Processing CreateTablet for tablet f465d0bb46674763abfc8d93f45cd13d (DEFAULT_TABLE table=default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.440158 1154 tablet_bootstrap.cc:492] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.440963 746 tablet_service.cc:1468] Processing CreateTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 (DEFAULT_TABLE table=default.Table_f9e7762f7fba4fc59cd1dd954b009b6b [id=d342099d88ee4e3ebbc5d5a1f17ba4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.441299 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed1ee73bb8f4761806a66f988b57bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.441344 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f465d0bb46674763abfc8d93f45cd13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.441427 738 tablet_service.cc:1468] Processing CreateTablet for tablet f465d0bb46674763abfc8d93f45cd13d (DEFAULT_TABLE table=default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.441717 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f465d0bb46674763abfc8d93f45cd13d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.443068 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1c995ce14c9a4966b842fcba520cf87d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c995ce14c9a4966b842fcba520cf87d already in progress: creating tablet
I20250629 01:51:46.444221 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c5771fdd618e45118df2b6dab7f668d9 (DEFAULT_TABLE table=default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.444586 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5771fdd618e45118df2b6dab7f668d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.448405 755 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.448596 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.448634 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 with delete_type TABLET_DATA_DELETED (replaced by tablet be8e0946a20340b38835c4d8daeaa7b8) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.448822 1158 tablet_replica.cc:331] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.448822 742 tablet_service.cc:1468] Processing CreateTablet for tablet c5771fdd618e45118df2b6dab7f668d9 (DEFAULT_TABLE table=default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.449044 1158 raft_consensus.cc:2241] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.449191 1158 raft_consensus.cc:2270] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.449222 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5771fdd618e45118df2b6dab7f668d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.452136 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 (DEFAULT_TABLE table=default.Table_2d471323a04d4c0389ad1d253aa5cd36 [id=f364113e01a74ae7bc5ff75396e88d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.452478 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca02bcd9467496c9057b3de9c1b1725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.458601 741 tablet_service.cc:1468] Processing CreateTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 (DEFAULT_TABLE table=default.Table_2d471323a04d4c0389ad1d253aa5cd36 [id=f364113e01a74ae7bc5ff75396e88d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.459863 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca02bcd9467496c9057b3de9c1b1725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.460055 1154 tablet_bootstrap.cc:654] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.461532 1158 ts_tablet_manager.cc:1905] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.467058 740 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.467124 1158 ts_tablet_manager.cc:1918] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.467233 1158 log.cc:1199] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f1e0de66406e4ce89ce56fa447fe9895
I20250629 01:51:46.467525 1158 ts_tablet_manager.cc:1939] T f1e0de66406e4ce89ce56fa447fe9895 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.468309 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f1e0de66406e4ce89ce56fa447fe9895 (table default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]) successfully deleted
I20250629 01:51:46.478111 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.478607 1622 catalog_manager.cc:1261] Loaded cluster ID: 3bd8135387974b63b91ae240fae34427
I20250629 01:51:46.478757 1622 catalog_manager.cc:1554] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: loading cluster ID for follower catalog manager: success
W20250629 01:51:46.481444 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 already in progress: creating tablet
W20250629 01:51:46.482126 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2ec92eddfabf40918a6cabaff5fe113a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ec92eddfabf40918a6cabaff5fe113a already in progress: creating tablet
W20250629 01:51:46.482728 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 already in progress: creating tablet
W20250629 01:51:46.483287 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d1a8a9bc691e4b158276f34a51c0dc61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1a8a9bc691e4b158276f34a51c0dc61 already in progress: creating tablet
I20250629 01:51:46.484639 1155 tablet_bootstrap.cc:654] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.487303 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7b6e3def9c71407cbd04500227e036ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.487509 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_49846b0d606c4bf28681117f0a8340bd" } new_table_name: "default.Table_fadb5cdf8c8944018b9259b5fc865dec" modify_external_catalogs: true
W20250629 01:51:46.487776 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b6e3def9c71407cbd04500227e036ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.490420 1155 tablet_bootstrap.cc:492] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.490562 1155 ts_tablet_manager.cc:1397] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.194s user 0.003s sys 0.000s
I20250629 01:51:46.491266 1155 raft_consensus.cc:357] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.491436 1155 raft_consensus.cc:383] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.491521 1155 raft_consensus.cc:738] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
W20250629 01:51:46.497589 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8976cca816b442cc9a7cb4f23cc13016 consensus state has no leader
I20250629 01:51:46.500715 1018 tablet_service.cc:1468] Processing CreateTablet for tablet a43f6594e4844af580c99ea598272aae (DEFAULT_TABLE table=default.Table_df0fc31afd154b05a7d3d84bec25e140 [id=c1135b07e9bd455cb21ae0061c94e017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.501091 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43f6594e4844af580c99ea598272aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.501188 744 tablet_service.cc:1468] Processing CreateTablet for tablet a43f6594e4844af580c99ea598272aae (DEFAULT_TABLE table=default.Table_df0fc31afd154b05a7d3d84bec25e140 [id=c1135b07e9bd455cb21ae0061c94e017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.501561 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43f6594e4844af580c99ea598272aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.504314 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 (DEFAULT_TABLE table=default.Table_82be402325034e178ca2423c9729a2dd [id=90dec8a37f534ef4a43c4b9c3bd97d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.504701 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b (DEFAULT_TABLE table=default.Table_d11d1c20ae434e399df06213b6217417 [id=e0abed95afd24fc883d4f6e2d87b79f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.505223 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2045012c80d049eaabf7dec1df28b087 (DEFAULT_TABLE table=default.Table_d7a4536b4ffa4887a2b30a00f22275c6 [id=8d7ebfe144f84b28ac0f374fda7d0584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.505299 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 056eeff228974a8e9036a37b58a2a675 (DEFAULT_TABLE table=default.Table_bb5794f5710541f0adb81634de0ead93 [id=cd9d3f908ad4474cadbe36f9e2ba2cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.505616 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2045012c80d049eaabf7dec1df28b087. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.505731 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 (DEFAULT_TABLE table=default.Table_8a1e4394a537495b9bf3f9e4335f7936 [id=e6a1c068fbca47308ae288b6d28fc63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.505779 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c320a3e0f5fb4ee782878b813157b1cb" } new_table_name: "default.Table_c08e69cfa0894f67beafee56aa098ad9" modify_external_catalogs: true
I20250629 01:51:46.506382 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 2045012c80d049eaabf7dec1df28b087 (DEFAULT_TABLE table=default.Table_d7a4536b4ffa4887a2b30a00f22275c6 [id=8d7ebfe144f84b28ac0f374fda7d0584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.506692 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2045012c80d049eaabf7dec1df28b087. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.506829 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_93247e23895942578dd7e5f95a2cd9e4" } new_table_name: "default.Table_0400c31cb3764210a6e2492c308e02e8" modify_external_catalogs: true
I20250629 01:51:46.508108 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33aa0a66cdbf49c5820e2b43b021f28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.508042 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_62271eae9afc4deca5dfc76ed3c291c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.508498 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62271eae9afc4deca5dfc76ed3c291c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.510751 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056eeff228974a8e9036a37b58a2a675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.511183 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557ecee0d6e84f77ad10bbdd27859ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.512321 749 tablet_service.cc:1468] Processing CreateTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b (DEFAULT_TABLE table=default.Table_d11d1c20ae434e399df06213b6217417 [id=e0abed95afd24fc883d4f6e2d87b79f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.512717 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33aa0a66cdbf49c5820e2b43b021f28b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.513165 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add482b533d045e7a3010064fae730cf consensus state has no leader
I20250629 01:51:46.515240 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_39a7d1cee49d4af2ab7db17efbf44268" } new_table_name: "default.Table_4e176f09e49f499982f892aaaeb1d681" modify_external_catalogs: true
I20250629 01:51:46.517108 742 tablet_service.cc:1468] Processing CreateTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 (DEFAULT_TABLE table=default.Table_82be402325034e178ca2423c9729a2dd [id=90dec8a37f534ef4a43c4b9c3bd97d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.517755 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557ecee0d6e84f77ad10bbdd27859ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.518133 746 tablet_service.cc:1468] Processing CreateTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 (DEFAULT_TABLE table=default.Table_8a1e4394a537495b9bf3f9e4335f7936 [id=e6a1c068fbca47308ae288b6d28fc63d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:46.518271 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bdc6a26b3594d6fb33a27cac5b29f4f consensus state has no leader
I20250629 01:51:46.518455 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e9646c3ad04ed580f1d3402010dd38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.519137 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e9646c3ad04ed580f1d3402010dd38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.521554 751 tablet_service.cc:1468] Processing CreateTablet for tablet 056eeff228974a8e9036a37b58a2a675 (DEFAULT_TABLE table=default.Table_bb5794f5710541f0adb81634de0ead93 [id=cd9d3f908ad4474cadbe36f9e2ba2cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.521900 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056eeff228974a8e9036a37b58a2a675. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.523958 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01d95356c7a7419bb56ddf67d451eadc consensus state has no leader
I20250629 01:51:46.525725 740 tablet_service.cc:1468] Processing CreateTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 (DEFAULT_TABLE table=default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0 [id=c51af262900b4b79ac1eeabd0619f61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.526095 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02bd46ceac1e4766ada4c4d2e95fda00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.532305 741 tablet_service.cc:1468] Processing CreateTablet for tablet 9b54228f0013424588a355ceb17d9784 (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.532330 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 9b54228f0013424588a355ceb17d9784 (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.532644 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b54228f0013424588a355ceb17d9784. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.532755 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b54228f0013424588a355ceb17d9784. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.534269 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 (DEFAULT_TABLE table=default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0 [id=c51af262900b4b79ac1eeabd0619f61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.534627 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02bd46ceac1e4766ada4c4d2e95fda00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.545621 1154 tablet_bootstrap.cc:492] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.545783 1154 ts_tablet_manager.cc:1397] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.106s user 0.002s sys 0.002s
I20250629 01:51:46.546509 1154 raft_consensus.cc:357] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.546715 1154 raft_consensus.cc:383] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.546742 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 0467b153057f491e81681fa399a2a1a9 (DEFAULT_TABLE table=default.Table_7b6e3def9c71407cbd04500227e036ad [id=d885f36a9fc24af7aba394f839958d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.546903 1154 raft_consensus.cc:738] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.547098 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0467b153057f491e81681fa399a2a1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.547665 1155 consensus_queue.cc:260] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.548430 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0467b153057f491e81681fa399a2a1a9 (DEFAULT_TABLE table=default.Table_7b6e3def9c71407cbd04500227e036ad [id=d885f36a9fc24af7aba394f839958d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.548708 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0467b153057f491e81681fa399a2a1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.547099 1154 consensus_queue.cc:260] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.552318 1155 ts_tablet_manager.cc:1428] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20250629 01:51:46.554016 1155 tablet_bootstrap.cc:492] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.555718 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c2591c65d53449c8801778606f1f05f1 (DEFAULT_TABLE table=default.Table_62271eae9afc4deca5dfc76ed3c291c2 [id=6619b075d5054667b7718e91265aea37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.556222 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2591c65d53449c8801778606f1f05f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.560031 1155 tablet_bootstrap.cc:654] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.558048 1154 ts_tablet_manager.cc:1428] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250629 01:51:46.564026 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.564306 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 7748846a3df14ffa801321ccde86e35f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.564555 1156 tablet_replica.cc:331] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.564741 1156 raft_consensus.cc:2241] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.564934 1158 tablet_replica.cc:331] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.564939 1156 raft_consensus.cc:2270] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.565101 1158 raft_consensus.cc:2241] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.565260 1158 raft_consensus.cc:2270] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.566042 1158 ts_tablet_manager.cc:1905] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.566007 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ba33f4a793dd4e44820144e75cdceeb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.566350 1156 ts_tablet_manager.cc:1905] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:46.566481 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba33f4a793dd4e44820144e75cdceeb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.564813 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b97cdfd111da4222af07e939cf5eae47" } modify_external_catalogs: true
I20250629 01:51:46.567365 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b97cdfd111da4222af07e939cf5eae47" } modify_external_catalogs: true
I20250629 01:51:46.569324 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_307dc406b50143efa856b28a689b1fe1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.569753 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_307dc406b50143efa856b28a689b1fe1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.570173 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a3f4932fdc864c30a60546170db190a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.570559 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3f4932fdc864c30a60546170db190a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.571947 1156 ts_tablet_manager.cc:1918] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.572046 1156 log.cc:1199] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/7748846a3df14ffa801321ccde86e35f
I20250629 01:51:46.572389 1156 ts_tablet_manager.cc:1939] T 7748846a3df14ffa801321ccde86e35f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.573872 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 7748846a3df14ffa801321ccde86e35f (table default.Table_9edf5b31c7b24325930accee8cb627c3 [id=79a016a743f641158a9240b362c274c3]) successfully deleted
I20250629 01:51:46.575160 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet eb26725bd175429d805947c4eda6e40a
I20250629 01:51:46.573921 1158 ts_tablet_manager.cc:1918] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.575477 1158 log.cc:1199] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/299922fa5e694bdf96f29f1522068656
I20250629 01:51:46.575878 1158 ts_tablet_manager.cc:1939] T 299922fa5e694bdf96f29f1522068656 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.576531 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ea26ca313df64a08bdf66b12c7b74613" } new_table_name: "default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a" modify_external_catalogs: true
I20250629 01:51:46.577105 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 299922fa5e694bdf96f29f1522068656 (table default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 [id=28e5371a950e42f8803d96b0ffa26cf8]) successfully deleted
I20250629 01:51:46.577896 1364 raft_consensus.cc:491] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:46.578035 1364 raft_consensus.cc:513] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.579042 1364 leader_election.cc:290] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:46.579365 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:46.579864 1622 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: acquiring CA information for follower catalog manager: success
W20250629 01:51:46.579869 986 leader_election.cc:343] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:46.580881 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet eb26725bd175429d805947c4eda6e40a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.581125 742 tablet_service.cc:1515] Processing DeleteTablet for tablet eb26725bd175429d805947c4eda6e40a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:46.581619 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet eb26725bd175429d805947c4eda6e40a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb26725bd175429d805947c4eda6e40a already in progress: creating tablet
I20250629 01:51:46.581974 1154 tablet_bootstrap.cc:492] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.582093 744 tablet_service.cc:1468] Processing CreateTablet for tablet c2591c65d53449c8801778606f1f05f1 (DEFAULT_TABLE table=default.Table_62271eae9afc4deca5dfc76ed3c291c2 [id=6619b075d5054667b7718e91265aea37]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:46.583005 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet eb26725bd175429d805947c4eda6e40a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb26725bd175429d805947c4eda6e40a already in progress: creating tablet
I20250629 01:51:46.583242 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2591c65d53449c8801778606f1f05f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.584707 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_eab06a5f9cbc4248a1757e3b5518545b" } modify_external_catalogs: true
I20250629 01:51:46.584851 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_eab06a5f9cbc4248a1757e3b5518545b" } modify_external_catalogs: true
W20250629 01:51:46.586262 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a582d4f907f340179024c0377d3893e2 consensus state has no leader
I20250629 01:51:46.589375 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]). Expected version 1 got 0
I20250629 01:51:46.589983 889 tablet_service.cc:1468] Processing CreateTablet for tablet bb98aa9d365541969ea1dc70960760b2 (DEFAULT_TABLE table=default.Table_62002e6a41684d369e22fb34ba498d8f [id=631e5b6c35a7419a8963c0a4e468e439]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.590394 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb98aa9d365541969ea1dc70960760b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.591007 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a72c6040421642578fac307ea9914ef6
I20250629 01:51:46.593299 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a72c6040421642578fac307ea9914ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.593535 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a72c6040421642578fac307ea9914ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.595136 1158 tablet_replica.cc:331] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.595409 1158 raft_consensus.cc:2241] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.595587 1156 tablet_replica.cc:331] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.595793 1158 raft_consensus.cc:2270] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.596027 1156 raft_consensus.cc:2241] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.596560 1158 ts_tablet_manager.cc:1905] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.596760 1156 raft_consensus.cc:2270] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.599212 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c995ce14c9a4966b842fcba520cf87d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.599903 1156 ts_tablet_manager.cc:1905] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.599997 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ed0327b20c3440b9f184793c8e1ee7d" } modify_external_catalogs: true
I20250629 01:51:46.600244 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ed0327b20c3440b9f184793c8e1ee7d" } modify_external_catalogs: true
I20250629 01:51:46.596050 1154 tablet_bootstrap.cc:654] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.604169 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_209b7bc3f1de4919a043f2ab3c1c5d6a" } modify_external_catalogs: true
I20250629 01:51:46.604367 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_209b7bc3f1de4919a043f2ab3c1c5d6a" } modify_external_catalogs: true
I20250629 01:51:46.605242 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a75d7986152f427d88ed34a443128bc6
I20250629 01:51:46.605484 1155 tablet_bootstrap.cc:492] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.605616 1155 ts_tablet_manager.cc:1397] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20250629 01:51:46.606258 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a75d7986152f427d88ed34a443128bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.606565 892 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.609388 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a75d7986152f427d88ed34a443128bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.606374 1155 raft_consensus.cc:357] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.610157 1155 raft_consensus.cc:383] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.610268 1155 raft_consensus.cc:738] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.610532 1155 consensus_queue.cc:260] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.611032 1155 ts_tablet_manager.cc:1428] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:46.611361 1155 tablet_bootstrap.cc:492] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.612782 1155 tablet_bootstrap.cc:654] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.613672 877 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.615254 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 (DEFAULT_TABLE table=default.Table_f9e7762f7fba4fc59cd1dd954b009b6b [id=d342099d88ee4e3ebbc5d5a1f17ba4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.615973 883 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.616909 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_55fba18dbe1547d8929080823f907fcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.617350 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4a9dba523727484f82caba8da2c65600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.617511 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55fba18dbe1547d8929080823f907fcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.617769 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a9dba523727484f82caba8da2c65600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.621562 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a254b0317477416389b93521d375802b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.621763 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_324620c729034dc2bd24e044a719599f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.622009 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a254b0317477416389b93521d375802b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.622179 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_324620c729034dc2bd24e044a719599f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.622346 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 3a81e6f9711c4d398ae09f33a75566ae from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:46.623183 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ea3a09c86c72411e97c6d80a7c665599"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.623616 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea3a09c86c72411e97c6d80a7c665599 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.626787 1154 tablet_bootstrap.cc:492] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.626979 1154 ts_tablet_manager.cc:1397] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20250629 01:51:46.627987 1154 raft_consensus.cc:357] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.628192 1154 raft_consensus.cc:383] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.628259 1154 raft_consensus.cc:738] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.633484 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 79e61d9576f8414ebf0074d57cdfb1f5 with delete_type TABLET_DATA_DELETED (replaced by tablet e82b1352bb4d49769808dc402c85890f) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.635133 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55fba18dbe1547d8929080823f907fcc" } modify_external_catalogs: true
I20250629 01:51:46.635272 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55fba18dbe1547d8929080823f907fcc" } modify_external_catalogs: true
I20250629 01:51:46.635764 1622 catalog_manager.cc:1604] T 00000000000000000000000000000000 P f304077528b34c61857a659e5873631d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250629 01:51:46.636148 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ed1ee73bb8f4761806a66f988b57bf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.636611 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.638813 1156 ts_tablet_manager.cc:1918] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250629 01:51:46.639781 1156 log.cc:1199] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a72c6040421642578fac307ea9914ef6
I20250629 01:51:46.640515 1156 ts_tablet_manager.cc:1939] T a72c6040421642578fac307ea9914ef6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.642437 893 tablet_service.cc:1468] Processing CreateTablet for tablet 2045012c80d049eaabf7dec1df28b087 (DEFAULT_TABLE table=default.Table_d7a4536b4ffa4887a2b30a00f22275c6 [id=8d7ebfe144f84b28ac0f374fda7d0584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.642585 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 (DEFAULT_TABLE table=default.Table_2d471323a04d4c0389ad1d253aa5cd36 [id=f364113e01a74ae7bc5ff75396e88d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.643045 885 tablet_service.cc:1468] Processing CreateTablet for tablet a43f6594e4844af580c99ea598272aae (DEFAULT_TABLE table=default.Table_df0fc31afd154b05a7d3d84bec25e140 [id=c1135b07e9bd455cb21ae0061c94e017]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.643471 890 tablet_service.cc:1468] Processing CreateTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 (DEFAULT_TABLE table=default.Table_82be402325034e178ca2423c9729a2dd [id=90dec8a37f534ef4a43c4b9c3bd97d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.643899 880 tablet_service.cc:1468] Processing CreateTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 (DEFAULT_TABLE table=default.Table_8a1e4394a537495b9bf3f9e4335f7936 [id=e6a1c068fbca47308ae288b6d28fc63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.644341 886 tablet_service.cc:1468] Processing CreateTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b (DEFAULT_TABLE table=default.Table_d11d1c20ae434e399df06213b6217417 [id=e0abed95afd24fc883d4f6e2d87b79f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.644800 878 tablet_service.cc:1468] Processing CreateTablet for tablet 056eeff228974a8e9036a37b58a2a675 (DEFAULT_TABLE table=default.Table_bb5794f5710541f0adb81634de0ead93 [id=cd9d3f908ad4474cadbe36f9e2ba2cd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.645018 1155 tablet_bootstrap.cc:492] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.645134 1155 ts_tablet_manager.cc:1397] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.003s
I20250629 01:51:46.645211 888 tablet_service.cc:1468] Processing CreateTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 (DEFAULT_TABLE table=default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0 [id=c51af262900b4b79ac1eeabd0619f61d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.645615 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9b54228f0013424588a355ceb17d9784 (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.645795 452 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 361a69e40c70466ba01f5d406cca14a7
I20250629 01:51:46.646381 1155 raft_consensus.cc:357] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.646600 894 tablet_service.cc:1468] Processing CreateTablet for tablet c2591c65d53449c8801778606f1f05f1 (DEFAULT_TABLE table=default.Table_62271eae9afc4deca5dfc76ed3c291c2 [id=6619b075d5054667b7718e91265aea37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.646652 1155 raft_consensus.cc:383] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.646747 1155 raft_consensus.cc:738] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.647073 1155 consensus_queue.cc:260] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.647653 1155 ts_tablet_manager.cc:1428] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:46.647997 1155 tablet_bootstrap.cc:492] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.649524 1155 tablet_bootstrap.cc:654] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.666333 1154 consensus_queue.cc:260] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.666816 1154 ts_tablet_manager.cc:1428] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250629 01:51:46.667188 887 tablet_service.cc:1468] Processing CreateTablet for tablet f465d0bb46674763abfc8d93f45cd13d (DEFAULT_TABLE table=default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.671253 884 tablet_service.cc:1468] Processing CreateTablet for tablet 0467b153057f491e81681fa399a2a1a9 (DEFAULT_TABLE table=default.Table_7b6e3def9c71407cbd04500227e036ad [id=d885f36a9fc24af7aba394f839958d79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.671641 1154 tablet_bootstrap.cc:492] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.671734 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2045012c80d049eaabf7dec1df28b087. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.672394 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0467b153057f491e81681fa399a2a1a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.673161 1154 tablet_bootstrap.cc:654] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.673465 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056eeff228974a8e9036a37b58a2a675. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.675438 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5e9646c3ad04ed580f1d3402010dd38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.675478 1158 ts_tablet_manager.cc:1918] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250629 01:51:46.675590 1158 log.cc:1199] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a72c6040421642578fac307ea9914ef6
I20250629 01:51:46.675931 1158 ts_tablet_manager.cc:1939] T a72c6040421642578fac307ea9914ef6 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.676811 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a72c6040421642578fac307ea9914ef6 (table default.Table_eab06a5f9cbc4248a1757e3b5518545b [id=6a41ce2e3dcf43ce9edb6392a244d9a2]) successfully deleted
I20250629 01:51:46.677796 1154 tablet_bootstrap.cc:492] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.677917 1154 ts_tablet_manager.cc:1397] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250629 01:51:46.678690 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:46.678902 1154 raft_consensus.cc:357] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.679028 1154 raft_consensus.cc:383] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.679085 1154 raft_consensus.cc:738] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.679271 1154 consensus_queue.cc:260] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:46.679821 1154 ts_tablet_manager.cc:1428] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:46.680331 1154 tablet_bootstrap.cc:492] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.681871 879 tablet_service.cc:1515] Processing DeleteTablet for tablet a72c6040421642578fac307ea9914ef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.682464 879 tablet_service.cc:1515] Processing DeleteTablet for tablet a75d7986152f427d88ed34a443128bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.685518 1154 tablet_bootstrap.cc:654] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.689595 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 36fcb819c61c4fabbb0da295d0002998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.689953 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:46.693498 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a75d7986152f427d88ed34a443128bc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a75d7986152f427d88ed34a443128bc6 already in progress: creating tablet
I20250629 01:51:46.694762 1158 tablet_replica.cc:331] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.695755 1157 ts_tablet_manager.cc:1918] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:46.695935 850 consensus_peers.cc:489] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:46.696254 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b070f6a145e04ae4b3bcf620e35d2943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.696547 988 leader_election.cc:343] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:46.696677 988 leader_election.cc:304] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:46.696779 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b070f6a145e04ae4b3bcf620e35d2943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.697031 1364 raft_consensus.cc:2747] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:46.701725 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2b15ded2e96e4eb5a060fa718fc40116" } new_table_name: "default.Table_e72b0bd01cbb4f8eaa5caecb2d75133a" modify_external_catalogs: true
I20250629 01:51:46.704710 883 tablet_service.cc:1468] Processing CreateTablet for tablet c5771fdd618e45118df2b6dab7f668d9 (DEFAULT_TABLE table=default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.705319 1157 log.cc:1199] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/6128463cd85d4f0cbfe21b0e5bd0e470
I20250629 01:51:46.708660 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33aa0a66cdbf49c5820e2b43b021f28b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.708891 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557ecee0d6e84f77ad10bbdd27859ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.709501 891 tablet_service.cc:1515] Processing DeleteTablet for tablet eb26725bd175429d805947c4eda6e40a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.709558 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2591c65d53449c8801778606f1f05f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.710137 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5771fdd618e45118df2b6dab7f668d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.711261 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02bd46ceac1e4766ada4c4d2e95fda00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.712932 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca02bcd9467496c9057b3de9c1b1725. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.713441 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b54228f0013424588a355ceb17d9784. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:46.714072 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0ca04e5f6b42df8567deb53a3e5f53 consensus state has no leader
I20250629 01:51:46.714371 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a43f6594e4844af580c99ea598272aae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.702684 1158 raft_consensus.cc:2241] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.715471 1159 tablet_bootstrap.cc:492] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:46.715384 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7acc3d53b2754aef87214be4a094b1f8" } new_table_name: "default.Table_614ed7de687c4876868bd54af35e3f8a" modify_external_catalogs: true
I20250629 01:51:46.717897 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a72c6040421642578fac307ea9914ef6 (table default.Table_eab06a5f9cbc4248a1757e3b5518545b [id=6a41ce2e3dcf43ce9edb6392a244d9a2]) successfully deleted
I20250629 01:51:46.720243 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ca0166de4e164756a7d5c1c2972df9a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.720718 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca0166de4e164756a7d5c1c2972df9a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.721935 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_da45cfc017de463a9a4b04db4cf3f710" } new_table_name: "default.Table_4226edee39d24dafb4dac54e8c6c7c58" modify_external_catalogs: true
W20250629 01:51:46.722663 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d87639255ec46d98fcfb0f9638ee39a consensus state has no leader
I20250629 01:51:46.725667 739 tablet_service.cc:1468] Processing CreateTablet for tablet e119711a47f14100a9455a1afd84b6d1 (DEFAULT_TABLE table=default.Table_ba33f4a793dd4e44820144e75cdceeb4 [id=5ba5e48e7fd145bf80961b241679641e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.726188 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e119711a47f14100a9455a1afd84b6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.726161 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_b39b700fff9245d8a067319ab01e5d8e [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 5 got 0
I20250629 01:51:46.726281 1156 tablet_replica.cc:331] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.726501 1156 raft_consensus.cc:2241] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.726781 1156 raft_consensus.cc:2270] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.727749 1156 ts_tablet_manager.cc:1905] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:46.728662 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a75d7986152f427d88ed34a443128bc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a75d7986152f427d88ed34a443128bc6 already in progress: creating tablet
I20250629 01:51:46.729228 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e3f08876e6774162a6aff102cc0e7bc9" } new_table_name: "default.Table_33cae964f7ab41198ea114d25de7742f" modify_external_catalogs: true
I20250629 01:51:46.731721 1157 ts_tablet_manager.cc:1939] T 6128463cd85d4f0cbfe21b0e5bd0e470 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:46.732043 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f465d0bb46674763abfc8d93f45cd13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.732937 1159 ts_tablet_manager.cc:1397] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.534s user 0.006s sys 0.011s
W20250629 01:51:46.737442 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 943f584fb9554fa4b65e5302443cec92 consensus state has no leader
I20250629 01:51:46.739210 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3a81e6f9711c4d398ae09f33a75566ae
I20250629 01:51:46.739614 446 catalog_manager.cc:6652] ReplaceTablet: tablet 3a81e6f9711c4d398ae09f33a75566ae deleted and replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb
I20250629 01:51:46.740548 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:51:46.741370 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 436) took 1516 ms. Trace:
I20250629 01:51:46.741590 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae with delete_type TABLET_DATA_DELETED (replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.742193 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.742281 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae with delete_type TABLET_DATA_DELETED (replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.743098 1156 ts_tablet_manager.cc:1918] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.743214 1156 log.cc:1199] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/79e61d9576f8414ebf0074d57cdfb1f5
I20250629 01:51:46.743539 1156 ts_tablet_manager.cc:1939] T 79e61d9576f8414ebf0074d57cdfb1f5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:46.744227 850 consensus_peers.cc:489] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Couldn't send request to peer ef5c5979bb9e475f983af7c3e9e0e019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a72c6040421642578fac307ea9914ef6. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:46.744381 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 79e61d9576f8414ebf0074d57cdfb1f5 (table default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]) successfully deleted
I20250629 01:51:46.744482 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.749562 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.753239 1158 raft_consensus.cc:2270] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.754757 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e119711a47f14100a9455a1afd84b6d1 (DEFAULT_TABLE table=default.Table_ba33f4a793dd4e44820144e75cdceeb4 [id=5ba5e48e7fd145bf80961b241679641e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.755162 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e119711a47f14100a9455a1afd84b6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.755813 1157 rpcz_store.cc:276] 0629 01:51:45.215851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.216445 (+ 594us) service_pool.cc:225] Handling call
0629 01:51:46.256607 (+1040162us) ts_tablet_manager.cc:1145] Deleting tablet 6128463cd85d4f0cbfe21b0e5bd0e470
0629 01:51:46.256615 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.564126 (+307511us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.648296 (+ 84170us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:46.732007 (+ 83711us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039979}
I20250629 01:51:46.754129 1158 ts_tablet_manager.cc:1905] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.760077 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 6128463cd85d4f0cbfe21b0e5bd0e470 (table default.Table_534d46b1a5ee4435844d4dd7c6626d1f [id=ff73871a84f745e383b03505bf30192d]) successfully deleted
W20250629 01:51:46.763911 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0323947b135a4308903b1eed1ca4db71 consensus state has no leader
I20250629 01:51:46.766232 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 439) took 1454 ms. Trace:
I20250629 01:51:46.767204 1157 rpcz_store.cc:276] 0629 01:51:45.306229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.307046 (+ 817us) service_pool.cc:225] Handling call
0629 01:51:46.756054 (+1449008us) ts_tablet_manager.cc:1145] Deleting tablet 1b0782c7e652419096710563930f2072
0629 01:51:46.756062 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.760610 (+ 4548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448820}
I20250629 01:51:46.767573 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 440) took 1461 ms. Trace:
I20250629 01:51:46.767870 1158 ts_tablet_manager.cc:1918] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.767989 1158 log.cc:1199] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/36fcb819c61c4fabbb0da295d0002998
W20250629 01:51:46.768502 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3a81e6f9711c4d398ae09f33a75566ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a81e6f9711c4d398ae09f33a75566ae already in progress: creating tablet
W20250629 01:51:46.768508 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1b0782c7e652419096710563930f2072 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b0782c7e652419096710563930f2072 already in progress: creating tablet
W20250629 01:51:46.768898 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 327c303b6b9347f5b4cb78f74d41a44a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 327c303b6b9347f5b4cb78f74d41a44a already in progress: creating tablet
I20250629 01:51:46.769773 1155 tablet_bootstrap.cc:492] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.769960 1155 ts_tablet_manager.cc:1397] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.010s
I20250629 01:51:46.770627 1155 raft_consensus.cc:357] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.770800 1155 raft_consensus.cc:383] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.770897 1155 raft_consensus.cc:738] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.771209 1155 consensus_queue.cc:260] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.771683 1155 ts_tablet_manager.cc:1428] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:46.772372 1155 tablet_bootstrap.cc:492] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.774344 1155 tablet_bootstrap.cc:654] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.777647 739 tablet_service.cc:1515] Processing DeleteTablet for tablet dfee69f9484046d8abe10d2f07033e09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.778218 1156 tablet_replica.cc:331] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.778402 1156 raft_consensus.cc:2241] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.778556 1156 raft_consensus.cc:2270] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.778632 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_913feb22c480457e96599a6814b30c60" } modify_external_catalogs: true
I20250629 01:51:46.778915 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_913feb22c480457e96599a6814b30c60" } modify_external_catalogs: true
I20250629 01:51:46.781454 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_57c0327ef1554e008fc2b4cbc62b28f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.782272 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57c0327ef1554e008fc2b4cbc62b28f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.783046 1157 rpcz_store.cc:276] 0629 01:51:45.306361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.307262 (+ 901us) service_pool.cc:225] Handling call
0629 01:51:46.767460 (+1460198us) ts_tablet_manager.cc:1145] Deleting tablet cabb0298771e4fd5be29e530d3a8ed3d
0629 01:51:46.767468 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.767544 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1460040}
I20250629 01:51:46.783995 1156 ts_tablet_manager.cc:1905] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.787041 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_87ba700594ec433ebf0252c79cdbdab9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.787469 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87ba700594ec433ebf0252c79cdbdab9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.787971 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cabb0298771e4fd5be29e530d3a8ed3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cabb0298771e4fd5be29e530d3a8ed3d already in progress: creating tablet
I20250629 01:51:46.788709 1154 tablet_bootstrap.cc:492] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.789052 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ed7d7f8855434f968c91b5a593527fc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.789142 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_223fe61dd1034f42a871c3ff477bcf1e" } new_table_name: "default.Table_d759086c226f4bb0a9aa12de777505e2" modify_external_catalogs: true
W20250629 01:51:46.789479 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed7d7f8855434f968c91b5a593527fc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.792259 1154 ts_tablet_manager.cc:1397] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.112s user 0.008s sys 0.006s
I20250629 01:51:46.793516 1154 raft_consensus.cc:357] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.793792 1154 raft_consensus.cc:383] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.793913 1154 raft_consensus.cc:738] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.794486 1154 consensus_queue.cc:260] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.795166 1154 ts_tablet_manager.cc:1428] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:46.795406 1154 tablet_bootstrap.cc:492] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.798316 891 tablet_service.cc:1468] Processing CreateTablet for tablet e119711a47f14100a9455a1afd84b6d1 (DEFAULT_TABLE table=default.Table_ba33f4a793dd4e44820144e75cdceeb4 [id=5ba5e48e7fd145bf80961b241679641e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.799669 1154 tablet_bootstrap.cc:654] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.804622 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e119711a47f14100a9455a1afd84b6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.804662 1159 raft_consensus.cc:357] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.804802 1159 raft_consensus.cc:383] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.805415 1159 raft_consensus.cc:738] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.805619 1159 consensus_queue.cc:260] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:46.807700 1159 ts_tablet_manager.cc:1428] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.073s user 0.001s sys 0.001s
I20250629 01:51:46.807657 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_49d0e0f59d084d388696b7b11d6d50d7" } modify_external_catalogs: true
I20250629 01:51:46.807783 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c7d1c2fe02d04423803f16f69f0d6625" } modify_external_catalogs: true
I20250629 01:51:46.807945 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c7d1c2fe02d04423803f16f69f0d6625" } modify_external_catalogs: true
I20250629 01:51:46.807814 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_49d0e0f59d084d388696b7b11d6d50d7" } modify_external_catalogs: true
I20250629 01:51:46.809316 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 441) took 1482 ms. Trace:
I20250629 01:51:46.809202 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae with delete_type TABLET_DATA_DELETED (replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.809772 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6092e19acf974f2dad5a903da72944ba
I20250629 01:51:46.810592 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6092e19acf974f2dad5a903da72944ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.810779 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 6092e19acf974f2dad5a903da72944ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.815220 1158 ts_tablet_manager.cc:1939] T 36fcb819c61c4fabbb0da295d0002998 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.818945 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 36fcb819c61c4fabbb0da295d0002998 (table default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]) successfully deleted
I20250629 01:51:46.820425 1158 tablet_replica.cc:331] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:46.820665 1159 tablet_bootstrap.cc:492] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:46.820947 1158 raft_consensus.cc:2241] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.821115 1158 raft_consensus.cc:2270] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.822187 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6092e19acf974f2dad5a903da72944ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.826220 1158 ts_tablet_manager.cc:1905] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.817813 1157 rpcz_store.cc:276] 0629 01:51:45.306481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.307445 (+ 964us) service_pool.cc:225] Handling call
0629 01:51:46.783652 (+1476207us) ts_tablet_manager.cc:1145] Deleting tablet c0b9afb9976541b2bc7ab188f35a62ec
0629 01:51:46.783767 (+ 115us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.788483 (+ 4716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476069}
W20250629 01:51:46.826810 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c0b9afb9976541b2bc7ab188f35a62ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0b9afb9976541b2bc7ab188f35a62ec already in progress: creating tablet
I20250629 01:51:46.830612 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e33e44d3bb7d4104866f13a3bd938c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.831277 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e33e44d3bb7d4104866f13a3bd938c15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.833948 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a12fffe61d7140308acd181bfa3d2700 consensus state has no leader
I20250629 01:51:46.834708 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b50cf2cfa18641428019543c0982188d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:46.834823 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.835110 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b50cf2cfa18641428019543c0982188d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:46.835322 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.836300 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0a1d504b9fd444a38d8e74bd4b2efd44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.836645 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a1d504b9fd444a38d8e74bd4b2efd44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.840607 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_93dfa58f836843938471c38606d8e6b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.840994 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93dfa58f836843938471c38606d8e6b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.843041 1681 tablet.cc:1722] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:46.843515 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 442) took 1530 ms. Trace:
I20250629 01:51:46.843286 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c712d978a3848b9bb3abdbce8314dec) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.844400 1157 rpcz_store.cc:276] 0629 01:51:45.306565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.307601 (+ 1036us) service_pool.cc:225] Handling call
0629 01:51:46.826449 (+1518848us) ts_tablet_manager.cc:1145] Deleting tablet 5b8a2d1a7f7146f4968590914374ba23
0629 01:51:46.832519 (+ 6070us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.837070 (+ 4551us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518702}
I20250629 01:51:46.844771 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 443) took 1538 ms. Trace:
W20250629 01:51:46.845463 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5b8a2d1a7f7146f4968590914374ba23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b8a2d1a7f7146f4968590914374ba23 already in progress: creating tablet
I20250629 01:51:46.846891 1157 rpcz_store.cc:276] 0629 01:51:45.306649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.307773 (+ 1124us) service_pool.cc:225] Handling call
0629 01:51:46.844636 (+1536863us) ts_tablet_manager.cc:1145] Deleting tablet 6f3a65c9c1c740b1865b96a6e2aea524
0629 01:51:46.844644 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.844734 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536725}
I20250629 01:51:46.847275 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 444) took 1540 ms. Trace:
W20250629 01:51:46.847803 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6f3a65c9c1c740b1865b96a6e2aea524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f3a65c9c1c740b1865b96a6e2aea524 already in progress: creating tablet
I20250629 01:51:46.847879 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ed7d7f8855434f968c91b5a593527fc4" } modify_external_catalogs: true
I20250629 01:51:46.848007 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ed7d7f8855434f968c91b5a593527fc4" } modify_external_catalogs: true
I20250629 01:51:46.853969 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f465d0bb46674763abfc8d93f45cd13d
I20250629 01:51:46.855062 744 tablet_service.cc:1515] Processing DeleteTablet for tablet f465d0bb46674763abfc8d93f45cd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.855326 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet f465d0bb46674763abfc8d93f45cd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:46.858317 1156 ts_tablet_manager.cc:1918] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.858536 1156 log.cc:1199] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/dfee69f9484046d8abe10d2f07033e09
I20250629 01:51:46.858855 1156 ts_tablet_manager.cc:1939] T dfee69f9484046d8abe10d2f07033e09 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.859982 1159 tablet_bootstrap.cc:654] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.860265 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet dfee69f9484046d8abe10d2f07033e09 (table default.Table_2bd4de753ed04021b2e3122b263bafab [id=40b322522a664b8f8bb8f39388b8ddf6]) successfully deleted
W20250629 01:51:46.860466 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6092e19acf974f2dad5a903da72944ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6092e19acf974f2dad5a903da72944ba already in progress: creating tablet
W20250629 01:51:46.860646 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f465d0bb46674763abfc8d93f45cd13d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f465d0bb46674763abfc8d93f45cd13d already in progress: creating tablet
I20250629 01:51:46.860932 1155 tablet_bootstrap.cc:492] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:46.861275 1155 ts_tablet_manager.cc:1397] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.006s
I20250629 01:51:46.861940 1155 raft_consensus.cc:357] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.862258 1155 raft_consensus.cc:383] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.862341 1155 raft_consensus.cc:738] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.862548 1155 consensus_queue.cc:260] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.863302 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f465d0bb46674763abfc8d93f45cd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:46.867228 1158 ts_tablet_manager.cc:1918] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.867347 1158 log.cc:1199] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5f60c48c924d49a186a8c81d154bdc54
I20250629 01:51:46.867681 1158 ts_tablet_manager.cc:1939] T 5f60c48c924d49a186a8c81d154bdc54 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:46.867792 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.868019 1155 ts_tablet_manager.cc:1428] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:46.867131 1157 rpcz_store.cc:276] 0629 01:51:45.306796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.307945 (+ 1149us) service_pool.cc:225] Handling call
0629 01:51:46.847147 (+1539202us) ts_tablet_manager.cc:1145] Deleting tablet 4a692f00ba244a80b9a6d07029f8d27e
0629 01:51:46.847159 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.847237 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1539011}
W20250629 01:51:46.868314 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 299922fa5e694bdf96f29f1522068656 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 299922fa5e694bdf96f29f1522068656 already in progress: creating tablet
W20250629 01:51:46.868470 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4a692f00ba244a80b9a6d07029f8d27e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a692f00ba244a80b9a6d07029f8d27e already in progress: creating tablet
I20250629 01:51:46.868546 1682 tablet.cc:1722] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:46.872601 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5f60c48c924d49a186a8c81d154bdc54 (table default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]) successfully deleted
I20250629 01:51:46.873395 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 299922fa5e694bdf96f29f1522068656 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:46.873730 1156 tablet_replica.cc:331] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:46.873970 1156 raft_consensus.cc:2241] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.874174 1156 raft_consensus.cc:2270] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:46.875703 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3a81e6f9711c4d398ae09f33a75566ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a81e6f9711c4d398ae09f33a75566ae already in progress: creating tablet
W20250629 01:51:46.875993 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 327c303b6b9347f5b4cb78f74d41a44a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 327c303b6b9347f5b4cb78f74d41a44a already in progress: creating tablet
W20250629 01:51:46.876271 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6092e19acf974f2dad5a903da72944ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6092e19acf974f2dad5a903da72944ba already in progress: creating tablet
W20250629 01:51:46.876511 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f465d0bb46674763abfc8d93f45cd13d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f465d0bb46674763abfc8d93f45cd13d already in progress: creating tablet
I20250629 01:51:46.877763 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 445) took 1562 ms. Trace:
I20250629 01:51:46.878077 1156 ts_tablet_manager.cc:1905] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:46.879408 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ec0ca04e5f6b42df8567deb53a3e5f53 (table default.Table_e72b0bd01cbb4f8eaa5caecb2d75133a [id=4a83041e3c7349f7a0b628c27e3a2139], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ec0ca04e5f6b42df8567deb53a3e5f53 consensus state has no leader
I20250629 01:51:46.879935 1157 rpcz_store.cc:276] 0629 01:51:45.306930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.308161 (+ 1231us) service_pool.cc:225] Handling call
0629 01:51:46.868238 (+1560077us) ts_tablet_manager.cc:1145] Deleting tablet 5c6f3f19a4d441f58c8c6099e04f82ec
0629 01:51:46.868244 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:46.869023 (+ 779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559914}
I20250629 01:51:46.880254 1157 tablet_replica.cc:331] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:46.883397 1156 ts_tablet_manager.cc:1918] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:46.883502 1156 log.cc:1199] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/299922fa5e694bdf96f29f1522068656
I20250629 01:51:46.883811 1156 ts_tablet_manager.cc:1939] T 299922fa5e694bdf96f29f1522068656 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:46.884464 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 299922fa5e694bdf96f29f1522068656 (table default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 [id=28e5371a950e42f8803d96b0ffa26cf8]) successfully deleted
I20250629 01:51:46.884754 1155 tablet_bootstrap.cc:492] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:46.886175 1155 tablet_bootstrap.cc:654] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.887492 1154 tablet_bootstrap.cc:492] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.887624 1154 ts_tablet_manager.cc:1397] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.000s
I20250629 01:51:46.888355 1154 raft_consensus.cc:357] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.888517 1154 raft_consensus.cc:383] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:46.888598 1154 raft_consensus.cc:738] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:46.894292 1154 consensus_queue.cc:260] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:46.894879 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f08ef32d58e24a9c93fb34da05736d8d
I20250629 01:51:46.898491 1154 ts_tablet_manager.cc:1428] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250629 01:51:46.902333 1154 tablet_bootstrap.cc:492] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:46.902789 1157 raft_consensus.cc:2241] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:46.902957 1157 raft_consensus.cc:2270] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:46.903878 1157 ts_tablet_manager.cc:1905] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:46.906687 744 tablet_service.cc:1468] Processing CreateTablet for tablet 6b0ef1ba46ac487ab1502c3d116814e1 (DEFAULT_TABLE table=default.Table_b070f6a145e04ae4b3bcf620e35d2943 [id=ff485047c4b7465294c759c95fd81876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.907099 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0ef1ba46ac487ab1502c3d116814e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.910353 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_6316ef46c0a6445190d1fa69105eacaa [id=95ccd48727404118a0e2c26dd2103a12]). Expected version 1 got 0
I20250629 01:51:46.910902 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3827f711f3174f66b469302de8091583"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.911401 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3827f711f3174f66b469302de8091583 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.913203 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 6b0ef1ba46ac487ab1502c3d116814e1 (DEFAULT_TABLE table=default.Table_b070f6a145e04ae4b3bcf620e35d2943 [id=ff485047c4b7465294c759c95fd81876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.913538 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0ef1ba46ac487ab1502c3d116814e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.915714 887 tablet_service.cc:1468] Processing CreateTablet for tablet 3070ce8c467a47b0a833374450911291 (DEFAULT_TABLE table=default.Table_ea3a09c86c72411e97c6d80a7c665599 [id=17d7bb0246e9438ea55cf8ef9af10043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.916342 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 18058ac8424043388bf4569e33d1da75 (DEFAULT_TABLE table=default.Table_4a9dba523727484f82caba8da2c65600 [id=0309c0b43ff74420b08ee16e5ae6001d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.916618 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18058ac8424043388bf4569e33d1da75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.917196 1154 tablet_bootstrap.cc:654] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:46.917568 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe] alter complete (version 1)
I20250629 01:51:46.920310 1020 tablet_service.cc:1468] Processing CreateTablet for tablet d7cbc2e6148846f9a58239b089e5a052 (DEFAULT_TABLE table=default.Table_307dc406b50143efa856b28a689b1fe1 [id=fbf8a98def284f10bac083131481cafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.920605 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7cbc2e6148846f9a58239b089e5a052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.921468 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3070ce8c467a47b0a833374450911291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.921578 880 tablet_service.cc:1468] Processing CreateTablet for tablet 60732c68906b4805bfa63cc4dafb4ceb (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.921475 889 tablet_service.cc:1468] Processing CreateTablet for tablet aef865f51a6b4191b2ab453c15094cc1 (DEFAULT_TABLE table=default.Table_324620c729034dc2bd24e044a719599f [id=0862cdbfdd25494c939392fa5143c314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.924237 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60732c68906b4805bfa63cc4dafb4ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.924530 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8bb871e6aa1745a6a3007e6b15ee75cd" } new_table_name: "default.Table_b557e22870a142208875b21ca442341b" modify_external_catalogs: true
I20250629 01:51:46.925128 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e63b2128684f4cfc898dee216184ddfa" } new_table_name: "default.Table_27e248a6eaa04f5d8039135e8ff089e1" modify_external_catalogs: true
I20250629 01:51:46.925390 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef865f51a6b4191b2ab453c15094cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.925527 749 tablet_service.cc:1468] Processing CreateTablet for tablet faca0e904a04456d9aead23abd44e00a (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.925863 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faca0e904a04456d9aead23abd44e00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.925933 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9dad47c452ca45039f44d08c43a7cca3 (DEFAULT_TABLE table=default.Table_a254b0317477416389b93521d375802b [id=589acdc7797d4f598bfa908a3dce2e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.925918 739 tablet_service.cc:1468] Processing CreateTablet for tablet 18058ac8424043388bf4569e33d1da75 (DEFAULT_TABLE table=default.Table_4a9dba523727484f82caba8da2c65600 [id=0309c0b43ff74420b08ee16e5ae6001d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.926314 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18058ac8424043388bf4569e33d1da75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.928361 751 tablet_service.cc:1468] Processing CreateTablet for tablet 3070ce8c467a47b0a833374450911291 (DEFAULT_TABLE table=default.Table_ea3a09c86c72411e97c6d80a7c665599 [id=17d7bb0246e9438ea55cf8ef9af10043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.928519 741 tablet_service.cc:1468] Processing CreateTablet for tablet aef865f51a6b4191b2ab453c15094cc1 (DEFAULT_TABLE table=default.Table_324620c729034dc2bd24e044a719599f [id=0862cdbfdd25494c939392fa5143c314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.928659 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3070ce8c467a47b0a833374450911291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.928846 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef865f51a6b4191b2ab453c15094cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.929131 740 tablet_service.cc:1468] Processing CreateTablet for tablet 9dad47c452ca45039f44d08c43a7cca3 (DEFAULT_TABLE table=default.Table_a254b0317477416389b93521d375802b [id=589acdc7797d4f598bfa908a3dce2e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.929436 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dad47c452ca45039f44d08c43a7cca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.930980 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dad47c452ca45039f44d08c43a7cca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.932325 745 tablet_service.cc:1468] Processing CreateTablet for tablet c84dfb60f67548ec9fa4c84c42e500c8 (DEFAULT_TABLE table=default.Table_ca0166de4e164756a7d5c1c2972df9a7 [id=764852c141de44e480459c9576cb46d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.932610 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84dfb60f67548ec9fa4c84c42e500c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.932572 742 tablet_service.cc:1468] Processing CreateTablet for tablet d7cbc2e6148846f9a58239b089e5a052 (DEFAULT_TABLE table=default.Table_307dc406b50143efa856b28a689b1fe1 [id=fbf8a98def284f10bac083131481cafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.932870 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7cbc2e6148846f9a58239b089e5a052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.933043 746 tablet_service.cc:1468] Processing CreateTablet for tablet 60732c68906b4805bfa63cc4dafb4ceb (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.933372 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60732c68906b4805bfa63cc4dafb4ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.940493 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_12d6ac1cd45c43eeac32cc51d3aae591" } new_table_name: "default.Table_d108e95f413c4b36a6fc67336e812ffe" modify_external_catalogs: true
I20250629 01:51:46.940907 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b86276d79e6a4da3859cdc3c35b24164"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:46.941427 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b86276d79e6a4da3859cdc3c35b24164 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:46.944329 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_81df428605084d568b00885fe59d5e29" } modify_external_catalogs: true
I20250629 01:51:46.952149 1016 tablet_service.cc:1468] Processing CreateTablet for tablet faca0e904a04456d9aead23abd44e00a (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.958317 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faca0e904a04456d9aead23abd44e00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.958793 1013 tablet_service.cc:1468] Processing CreateTablet for tablet c84dfb60f67548ec9fa4c84c42e500c8 (DEFAULT_TABLE table=default.Table_ca0166de4e164756a7d5c1c2972df9a7 [id=764852c141de44e480459c9576cb46d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.959137 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84dfb60f67548ec9fa4c84c42e500c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.962689 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 3070ce8c467a47b0a833374450911291 (DEFAULT_TABLE table=default.Table_ea3a09c86c72411e97c6d80a7c665599 [id=17d7bb0246e9438ea55cf8ef9af10043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.963552 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9dad47c452ca45039f44d08c43a7cca3 (DEFAULT_TABLE table=default.Table_a254b0317477416389b93521d375802b [id=589acdc7797d4f598bfa908a3dce2e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.965253 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b39b700fff9245d8a067319ab01e5d8e" } new_table_name: "default.Table_f07aa6eb42b6478781eb5a65dea53212" modify_external_catalogs: true
I20250629 01:51:46.965559 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3070ce8c467a47b0a833374450911291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.966008 1154 tablet_bootstrap.cc:492] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:46.963441 1032 tablet_service.cc:1468] Processing CreateTablet for tablet aef865f51a6b4191b2ab453c15094cc1 (DEFAULT_TABLE table=default.Table_324620c729034dc2bd24e044a719599f [id=0862cdbfdd25494c939392fa5143c314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.966586 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dad47c452ca45039f44d08c43a7cca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.968142 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef865f51a6b4191b2ab453c15094cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.931242 885 tablet_service.cc:1468] Processing CreateTablet for tablet faca0e904a04456d9aead23abd44e00a (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.975839 886 tablet_service.cc:1468] Processing CreateTablet for tablet 18058ac8424043388bf4569e33d1da75 (DEFAULT_TABLE table=default.Table_4a9dba523727484f82caba8da2c65600 [id=0309c0b43ff74420b08ee16e5ae6001d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.986187 1159 tablet_bootstrap.cc:492] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:46.987094 1159 ts_tablet_manager.cc:1397] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.003s
I20250629 01:51:46.986213 883 tablet_service.cc:1468] Processing CreateTablet for tablet 6b0ef1ba46ac487ab1502c3d116814e1 (DEFAULT_TABLE table=default.Table_b070f6a145e04ae4b3bcf620e35d2943 [id=ff485047c4b7465294c759c95fd81876]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.988504 884 tablet_service.cc:1468] Processing CreateTablet for tablet c84dfb60f67548ec9fa4c84c42e500c8 (DEFAULT_TABLE table=default.Table_ca0166de4e164756a7d5c1c2972df9a7 [id=764852c141de44e480459c9576cb46d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.989248 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faca0e904a04456d9aead23abd44e00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.996126 1638 consensus_queue.cc:1035] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250629 01:51:46.996804 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_81df428605084d568b00885fe59d5e29" } modify_external_catalogs: true
I20250629 01:51:46.996856 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0d4320f615464fedbcab25137952f684
I20250629 01:51:46.998099 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 60732c68906b4805bfa63cc4dafb4ceb (DEFAULT_TABLE table=default.Table_eb9a6975124a4d049fa2c7043760c952 [id=c51c68dd349543d4a4a8bfb1b9aa1184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:46.998405 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60732c68906b4805bfa63cc4dafb4ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:46.999424 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4320f615464fedbcab25137952f684 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:46.999963 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0d4320f615464fedbcab25137952f684 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4320f615464fedbcab25137952f684 already in progress: creating tablet
I20250629 01:51:47.000159 1154 ts_tablet_manager.cc:1397] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.011s
I20250629 01:51:47.000901 1154 raft_consensus.cc:357] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.002899 1154 raft_consensus.cc:383] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.003015 1154 raft_consensus.cc:738] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.003285 1154 consensus_queue.cc:260] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.008018 1155 tablet_bootstrap.cc:492] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.008404 1154 ts_tablet_manager.cc:1428] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250629 01:51:47.009413 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 2ec92eddfabf40918a6cabaff5fe113a from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.011456 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f with delete_type TABLET_DATA_DELETED (replaced by tablet f898483ab9514132be8537ad03720097) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.011688 1158 tablet_replica.cc:331] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.011838 1158 raft_consensus.cc:2241] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.011998 1158 raft_consensus.cc:2270] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.012251 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18058ac8424043388bf4569e33d1da75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.012457 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b0ef1ba46ac487ab1502c3d116814e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.012606 1158 ts_tablet_manager.cc:1905] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.013387 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_53f9893ad8274a76ad3cea4fd31594c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.013813 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53f9893ad8274a76ad3cea4fd31594c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.014456 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d9cb1a00d22843f69edc93ba305abaad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.014959 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9cb1a00d22843f69edc93ba305abaad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.016232 1155 ts_tablet_manager.cc:1397] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.132s user 0.014s sys 0.007s
I20250629 01:51:47.017576 1158 ts_tablet_manager.cc:1918] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.017899 1154 tablet_bootstrap.cc:492] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.017895 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c70f3648ce00471bb5d6dbd7d12e6b50" } modify_external_catalogs: true
I20250629 01:51:47.018029 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c70f3648ce00471bb5d6dbd7d12e6b50" } modify_external_catalogs: true
I20250629 01:51:47.017676 1158 log.cc:1199] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/3ba0e267f48a4bf1b58fb2339694719f
I20250629 01:51:47.019307 1158 ts_tablet_manager.cc:1939] T 3ba0e267f48a4bf1b58fb2339694719f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.020146 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 3ba0e267f48a4bf1b58fb2339694719f (table default.Table_3abcbdb38a38484782a28269e7b69beb [id=29b0e64154664e928993c1035a1bb23c]) successfully deleted
I20250629 01:51:47.021936 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4320f615464fedbcab25137952f684 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.022068 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c84dfb60f67548ec9fa4c84c42e500c8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:47.022089 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2562570cb0846e19ebc97bdb43bd70d consensus state has no leader
I20250629 01:51:47.016924 1155 raft_consensus.cc:357] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.024423 1154 tablet_bootstrap.cc:654] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.024549 1155 raft_consensus.cc:383] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.024710 1155 raft_consensus.cc:738] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.025245 1155 consensus_queue.cc:260] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.026108 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4320f615464fedbcab25137952f684 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.026575 1155 ts_tablet_manager.cc:1428] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
W20250629 01:51:47.026628 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0d4320f615464fedbcab25137952f684 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4320f615464fedbcab25137952f684 already in progress: creating tablet
I20250629 01:51:47.028268 1155 tablet_bootstrap.cc:492] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.029990 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0f40aefcb7e14612bf2d435dfa37d66e
I20250629 01:51:47.030719 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f40aefcb7e14612bf2d435dfa37d66e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.030898 1158 tablet_replica.cc:331] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.031064 1158 raft_consensus.cc:2241] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.031224 1158 raft_consensus.cc:2270] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.031656 1158 ts_tablet_manager.cc:1905] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.018069 892 tablet_service.cc:1468] Processing CreateTablet for tablet d7cbc2e6148846f9a58239b089e5a052 (DEFAULT_TABLE table=default.Table_307dc406b50143efa856b28a689b1fe1 [id=fbf8a98def284f10bac083131481cafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.032837 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f40aefcb7e14612bf2d435dfa37d66e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.033028 1156 tablet_replica.cc:331] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:47.033193 1156 raft_consensus.cc:2241] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:47.033468 1156 raft_consensus.cc:2270] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.033928 1156 ts_tablet_manager.cc:1905] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.034070 1154 tablet_bootstrap.cc:492] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.034173 1154 ts_tablet_manager.cc:1397] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250629 01:51:47.034991 1154 raft_consensus.cc:357] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.035164 1154 raft_consensus.cc:383] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.035243 1154 raft_consensus.cc:738] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.035455 1154 consensus_queue.cc:260] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.038210 1158 ts_tablet_manager.cc:1918] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:47.038367 1158 log.cc:1199] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0f40aefcb7e14612bf2d435dfa37d66e
I20250629 01:51:47.038756 1158 ts_tablet_manager.cc:1939] T 0f40aefcb7e14612bf2d435dfa37d66e P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.039255 1154 ts_tablet_manager.cc:1428] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:47.039641 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]) successfully deleted
I20250629 01:51:47.041337 1154 tablet_bootstrap.cc:492] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.043793 1156 ts_tablet_manager.cc:1918] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:47.043949 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f40aefcb7e14612bf2d435dfa37d66e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.044000 1154 tablet_bootstrap.cc:654] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.045035 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7cbc2e6148846f9a58239b089e5a052. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.045370 1159 raft_consensus.cc:357] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.045878 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_93358968e4994116ad323cf7bdd275d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.046428 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93358968e4994116ad323cf7bdd275d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.048941 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c712d978a3848b9bb3abdbce8314dec) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.049180 1158 tablet_replica.cc:331] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.049366 1158 raft_consensus.cc:2241] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.049504 1158 raft_consensus.cc:2270] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.049748 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_40c6d676eac2406887e3b761d5357587" } new_table_name: "default.Table_ebbaf6a5d4294ece9bc2e803e056a2e7" modify_external_catalogs: true
I20250629 01:51:47.050657 1158 ts_tablet_manager.cc:1905] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.043994 1156 log.cc:1199] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0f40aefcb7e14612bf2d435dfa37d66e
I20250629 01:51:47.056017 1156 ts_tablet_manager.cc:1939] T 0f40aefcb7e14612bf2d435dfa37d66e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:47.056363 1159 raft_consensus.cc:383] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.056457 1159 raft_consensus.cc:738] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.056691 1159 consensus_queue.cc:260] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.056766 1158 ts_tablet_manager.cc:1918] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.056946 1158 log.cc:1199] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a2a9664bb7a946c48cf79a09955ecad5
I20250629 01:51:47.054478 751 tablet_service.cc:1515] Processing DeleteTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 with delete_type TABLET_DATA_DELETED (replaced by tablet be8e0946a20340b38835c4d8daeaa7b8) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.057046 1660 raft_consensus.cc:491] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.057152 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1e578e5fe38c40d38ba9b7759798a267" } modify_external_catalogs: true
I20250629 01:51:47.057171 1660 raft_consensus.cc:513] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.057286 1158 ts_tablet_manager.cc:1939] T a2a9664bb7a946c48cf79a09955ecad5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.057293 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1e578e5fe38c40d38ba9b7759798a267" } modify_external_catalogs: true
I20250629 01:51:47.057723 1660 leader_election.cc:290] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:47.058076 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a2a9664bb7a946c48cf79a09955ecad5 (table default.Table_33cae964f7ab41198ea114d25de7742f [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:47.058326 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:47.059043 712 leader_election.cc:343] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.059900 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:47.060081 1049 raft_consensus.cc:2466] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:47.060616 1154 tablet_bootstrap.cc:492] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.060745 1154 ts_tablet_manager.cc:1397] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.000s
W20250629 01:51:47.061735 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a792a26725954adea15ad57e31bfb224 consensus state has no leader
I20250629 01:51:47.061980 710 leader_election.cc:304] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:47.062269 1154 raft_consensus.cc:357] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.062549 1154 raft_consensus.cc:383] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.062611 1154 raft_consensus.cc:738] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.062817 1154 consensus_queue.cc:260] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.063192 1660 raft_consensus.cc:2802] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:47.063287 1660 raft_consensus.cc:491] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:47.063354 1660 raft_consensus.cc:3058] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.065608 1660 raft_consensus.cc:513] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.066243 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]) successfully deleted
I20250629 01:51:47.066906 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:47.067145 1049 raft_consensus.cc:3058] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.067170 1154 ts_tablet_manager.cc:1428] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:47.067279 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:47.067445 1156 tablet_replica.cc:331] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:47.067629 1156 raft_consensus.cc:2241] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.067786 1156 raft_consensus.cc:2270] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.069509 1049 raft_consensus.cc:2466] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:47.070477 1154 tablet_bootstrap.cc:492] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.071781 1154 tablet_bootstrap.cc:654] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:47.072439 712 leader_election.cc:343] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.072705 710 leader_election.cc:304] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:47.072995 1664 raft_consensus.cc:2802] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:47.073359 1157 ts_tablet_manager.cc:1918] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.073482 1157 log.cc:1199] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/fe14d1d54db74a99b05cf8b1a60827a9
I20250629 01:51:47.073802 1157 ts_tablet_manager.cc:1939] T fe14d1d54db74a99b05cf8b1a60827a9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:47.079484 1154 tablet_bootstrap.cc:492] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.079591 1154 ts_tablet_manager.cc:1397] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20250629 01:51:47.080379 1154 raft_consensus.cc:357] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.080518 1154 raft_consensus.cc:383] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.080579 1154 raft_consensus.cc:738] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.080804 1154 consensus_queue.cc:260] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.081507 1660 leader_election.cc:290] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:47.084499 1156 ts_tablet_manager.cc:1905] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.084985 1154 ts_tablet_manager.cc:1428] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250629 01:51:47.088929 1154 tablet_bootstrap.cc:492] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.089541 1664 raft_consensus.cc:695] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:47.089854 1664 consensus_queue.cc:237] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.090735 1154 tablet_bootstrap.cc:654] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.093740 1154 tablet_bootstrap.cc:492] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.093902 1154 ts_tablet_manager.cc:1397] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250629 01:51:47.093999 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 446) took 1771 ms. Trace:
I20250629 01:51:47.094460 1154 raft_consensus.cc:357] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.094619 1154 raft_consensus.cc:383] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.094682 1154 raft_consensus.cc:738] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.094836 1364 raft_consensus.cc:491] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.094960 1364 raft_consensus.cc:513] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.095436 1364 leader_election.cc:290] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:47.095996 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bedbb041e4094d8399de625badd971bb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.096138 1156 ts_tablet_manager.cc:1918] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.096236 1156 log.cc:1199] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f1e0de66406e4ce89ce56fa447fe9895
W20250629 01:51:47.096477 986 leader_election.cc:343] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.096551 1156 ts_tablet_manager.cc:1939] T f1e0de66406e4ce89ce56fa447fe9895 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:47.096902 1154 consensus_queue.cc:260] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.097338 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f1e0de66406e4ce89ce56fa447fe9895 (table default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]) successfully deleted
I20250629 01:51:47.101125 1154 ts_tablet_manager.cc:1428] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:47.102571 1157 rpcz_store.cc:276] 0629 01:51:45.310739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.311557 (+ 818us) service_pool.cc:225] Handling call
0629 01:51:46.880182 (+1568625us) ts_tablet_manager.cc:1145] Deleting tablet fe14d1d54db74a99b05cf8b1a60827a9
0629 01:51:46.880193 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.016991 (+136798us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.073273 (+ 56282us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.082060 (+ 8787us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7696,"tablet-delete.queue_time_us":1568464}
I20250629 01:51:47.103439 1157 tablet_replica.cc:331] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:47.105762 1154 tablet_bootstrap.cc:492] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.107098 1154 tablet_bootstrap.cc:654] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.109704 1157 raft_consensus.cc:2241] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.110677 1154 tablet_bootstrap.cc:492] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.110901 1154 ts_tablet_manager.cc:1397] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250629 01:51:47.111521 1154 raft_consensus.cc:357] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.111665 1154 raft_consensus.cc:383] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.111742 1154 raft_consensus.cc:738] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.111940 1154 consensus_queue.cc:260] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.114446 446 catalog_manager.cc:5582] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:47.114796 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_f07aa6eb42b6478781eb5a65dea53212 [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 6 got 0
I20250629 01:51:47.114985 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7e58c9f288004df185e961b29ae19383"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.115456 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e58c9f288004df185e961b29ae19383 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.116727 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.119220 1158 tablet_replica.cc:331] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.119379 1158 raft_consensus.cc:2241] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.119496 1158 raft_consensus.cc:2270] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.122104 1155 tablet_bootstrap.cc:654] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.123178 1157 raft_consensus.cc:2270] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.123301 1158 ts_tablet_manager.cc:1905] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:47.125638 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da93c730451d42bc8df1e582fb3af818 consensus state has no leader
I20250629 01:51:47.128680 1158 ts_tablet_manager.cc:1918] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.129743 1158 log.cc:1199] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:47.130219 1158 ts_tablet_manager.cc:1939] T 8bcc3f2a76bc4bae8b306b54e75656cc P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.131402 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8bcc3f2a76bc4bae8b306b54e75656cc (table default.Table_f6f27d5499af49359886879547b0ac31 [id=e1d26b8e31044413af1d4bddc58816e7]) successfully deleted
I20250629 01:51:47.132797 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:47.133033 1154 ts_tablet_manager.cc:1428] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.022s user 0.000s sys 0.001s
W20250629 01:51:47.133765 450 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet bedbb041e4094d8399de625badd971bb consensus state has no leader
I20250629 01:51:47.134917 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.135123 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.135282 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.136967 1158 tablet_replica.cc:331] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.137068 1157 ts_tablet_manager.cc:1905] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.137197 1158 raft_consensus.cc:2241] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.137379 1158 raft_consensus.cc:2270] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.140512 1154 tablet_bootstrap.cc:492] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.142618 1154 tablet_bootstrap.cc:654] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.144138 1158 ts_tablet_manager.cc:1905] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:47.149080 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b15300fd8ca7489f9d77f1f8992f8df3 consensus state has no leader
I20250629 01:51:47.149600 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 432680cae61946af9859760fe23110ec
I20250629 01:51:47.152065 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 432680cae61946af9859760fe23110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.152204 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 432680cae61946af9859760fe23110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:47.152714 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 432680cae61946af9859760fe23110ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 432680cae61946af9859760fe23110ec already in progress: creating tablet
I20250629 01:51:47.157691 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_56ce186b15474e3ebf35ac8f454778bf" } modify_external_catalogs: true
I20250629 01:51:47.157866 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_56ce186b15474e3ebf35ac8f454778bf" } modify_external_catalogs: true
I20250629 01:51:47.159153 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_21ecde86214e4cf4a564659ff667e485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.159214 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d2e4a152a81f434eb4a24f2dd01af367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.159436 1158 ts_tablet_manager.cc:1918] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.159521 1158 log.cc:1199] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/af9ab324ce7c488cbaf6e0f0322b5c58
W20250629 01:51:47.159729 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2e4a152a81f434eb4a24f2dd01af367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:47.159772 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21ecde86214e4cf4a564659ff667e485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.159821 1158 ts_tablet_manager.cc:1939] T af9ab324ce7c488cbaf6e0f0322b5c58 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.160720 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet af9ab324ce7c488cbaf6e0f0322b5c58 (table default.Table_2c04531a50c54791a64d6537b670d54f [id=e4025f53c2ea4b7aa8fafb057c102a3b]) successfully deleted
I20250629 01:51:47.160841 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9587986b83104f2aabf6ef98ebad9e37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.161273 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9587986b83104f2aabf6ef98ebad9e37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.161793 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_78723e0db17d407aaed94a045aac54c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.162263 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78723e0db17d407aaed94a045aac54c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:47.164013 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6b6d5ff92c343cb8f1b3039b84390af consensus state has no leader
I20250629 01:51:47.164178 1158 tablet_replica.cc:331] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.164273 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 4ed143377709465d9ae5df7c2ec3ca47 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.164352 1158 raft_consensus.cc:2241] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.164491 1158 raft_consensus.cc:2270] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.164976 1155 tablet_bootstrap.cc:492] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.165067 1158 ts_tablet_manager.cc:1905] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.165203 1155 ts_tablet_manager.cc:1397] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.137s user 0.011s sys 0.001s
I20250629 01:51:47.165882 1155 raft_consensus.cc:357] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.166091 1155 raft_consensus.cc:383] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.166215 1155 raft_consensus.cc:738] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.166450 1155 consensus_queue.cc:260] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.166942 1155 ts_tablet_manager.cc:1428] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:47.167239 1155 tablet_bootstrap.cc:492] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.172717 1155 tablet_bootstrap.cc:654] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.176342 1158 ts_tablet_manager.cc:1918] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.176468 1158 log.cc:1199] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4dbe70d4e4e74c6ba7db2766bd78c8ad
I20250629 01:51:47.176759 1158 ts_tablet_manager.cc:1939] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.178134 1154 tablet_bootstrap.cc:492] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.178263 1154 ts_tablet_manager.cc:1397] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.002s
I20250629 01:51:47.178390 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad (table default.Table_adc0a47bc8504a73aa8b9ede02097dd9 [id=604220139d4f4b31bfaca78aa2b32f29]) successfully deleted
I20250629 01:51:47.178869 1154 raft_consensus.cc:357] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.179035 1154 raft_consensus.cc:383] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.179100 1154 raft_consensus.cc:738] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.179332 1154 consensus_queue.cc:260] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.186715 1154 ts_tablet_manager.cc:1428] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250629 01:51:47.186749 1664 raft_consensus.cc:491] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.186998 1664 raft_consensus.cc:513] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.187626 1664 leader_election.cc:290] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:47.188378 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a9664bb7a946c48cf79a09955ecad5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:47.188772 916 raft_consensus.cc:2466] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:47.189347 712 leader_election.cc:304] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:51:47.190101 710 leader_election.cc:343] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a2a9664bb7a946c48cf79a09955ecad5
I20250629 01:51:47.190593 1664 raft_consensus.cc:2802] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:47.190832 1664 raft_consensus.cc:491] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:47.190991 1664 raft_consensus.cc:3058] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.193300 1664 raft_consensus.cc:513] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.196223 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 50d23f589de84dd5b150a17cee00c411
I20250629 01:51:47.197063 1154 tablet_bootstrap.cc:492] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.197710 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 50d23f589de84dd5b150a17cee00c411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.198434 1154 tablet_bootstrap.cc:654] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.195360 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a2a9664bb7a946c48cf79a09955ecad5 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c712d978a3848b9bb3abdbce8314dec) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.199273 1156 tablet_replica.cc:331] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:47.199441 1156 raft_consensus.cc:2241] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.199576 1156 raft_consensus.cc:2270] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.197379 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 50d23f589de84dd5b150a17cee00c411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.200026 1364 raft_consensus.cc:491] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.200193 1364 raft_consensus.cc:513] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.200759 1364 leader_election.cc:290] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:47.201118 710 leader_election.cc:343] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a2a9664bb7a946c48cf79a09955ecad5
I20250629 01:51:47.201155 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e508393b9cb044c4b90194c8aab3184a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:47.201573 986 leader_election.cc:343] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.201674 1664 leader_election.cc:290] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:47.202178 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8bcc3f2a76bc4bae8b306b54e75656cc because the tablet was not found. No further retry: Not found: Tablet not found: 8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:47.202348 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6b2a627b815e45c480dcaafe356a5d50" } modify_external_catalogs: true
I20250629 01:51:47.202428 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2a9664bb7a946c48cf79a09955ecad5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:47.202508 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6b2a627b815e45c480dcaafe356a5d50" } modify_external_catalogs: true
W20250629 01:51:47.202684 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 432680cae61946af9859760fe23110ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 432680cae61946af9859760fe23110ec already in progress: creating tablet
I20250629 01:51:47.202733 916 raft_consensus.cc:3058] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:47.203220 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 50d23f589de84dd5b150a17cee00c411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50d23f589de84dd5b150a17cee00c411 already in progress: creating tablet
I20250629 01:51:47.203734 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 432680cae61946af9859760fe23110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.204061 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 50d23f589de84dd5b150a17cee00c411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.204752 1156 ts_tablet_manager.cc:1905] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.204857 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bedbb041e4094d8399de625badd971bb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:47.205672 916 raft_consensus.cc:2466] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:47.205722 1154 tablet_bootstrap.cc:492] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.205822 1154 ts_tablet_manager.cc:1397] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20250629 01:51:47.205930 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dfa935fc642246518e47197bd7dda059"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.206094 712 leader_election.cc:304] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:47.206279 712 raft_consensus.cc:2684] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250629 01:51:47.206369 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfa935fc642246518e47197bd7dda059 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.206490 1154 raft_consensus.cc:357] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.206620 1154 raft_consensus.cc:383] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.206682 1154 raft_consensus.cc:738] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.206864 1154 consensus_queue.cc:260] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.206899 1690 tablet.cc:1722] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:47.207633 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e508393b9cb044c4b90194c8aab3184a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:47.207861 916 raft_consensus.cc:2466] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:47.208179 1697 raft_consensus.cc:491] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:47.209486 988 leader_election.cc:343] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.209611 988 leader_election.cc:304] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.210641 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e4d6f1eb43b24fda94aa0afcf4501195
I20250629 01:51:47.210597 1697 raft_consensus.cc:513] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.211217 1697 leader_election.cc:290] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:47.211350 1697 consensus_queue.cc:1035] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:47.212144 1364 raft_consensus.cc:2747] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.212338 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.212697 741 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.213289 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f60c48c924d49a186a8c81d154bdc54" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.213778 884 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.213816 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.209874 1157 ts_tablet_manager.cc:1918] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.214340 1157 log.cc:1199] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/dfee69f9484046d8abe10d2f07033e09
I20250629 01:51:47.214372 988 leader_election.cc:304] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.214473 1156 ts_tablet_manager.cc:1918] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.214582 1156 log.cc:1199] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a2a9664bb7a946c48cf79a09955ecad5
I20250629 01:51:47.214677 1157 ts_tablet_manager.cc:1939] T dfee69f9484046d8abe10d2f07033e09 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:47.214747 1364 raft_consensus.cc:2802] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:47.214834 1364 raft_consensus.cc:491] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:47.214910 1156 ts_tablet_manager.cc:1939] T a2a9664bb7a946c48cf79a09955ecad5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:47.214927 1364 raft_consensus.cc:3058] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:47.215106 850 leader_election.cc:343] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.215962 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a2a9664bb7a946c48cf79a09955ecad5 (table default.Table_33cae964f7ab41198ea114d25de7742f [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
W20250629 01:51:47.216543 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 50d23f589de84dd5b150a17cee00c411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50d23f589de84dd5b150a17cee00c411 already in progress: creating tablet
W20250629 01:51:47.217007 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e4d6f1eb43b24fda94aa0afcf4501195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4d6f1eb43b24fda94aa0afcf4501195 already in progress: creating tablet
I20250629 01:51:47.217633 1364 raft_consensus.cc:513] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.218127 1364 leader_election.cc:290] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.221035 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e508393b9cb044c4b90194c8aab3184a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:47.221063 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 458) took 1673 ms. Trace:
I20250629 01:51:47.221158 1157 rpcz_store.cc:276] 0629 01:51:45.547353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.547398 (+ 45us) service_pool.cc:225] Handling call
0629 01:51:47.102865 (+1555467us) ts_tablet_manager.cc:1145] Deleting tablet dfee69f9484046d8abe10d2f07033e09
0629 01:51:47.102875 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.177079 (+ 74204us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.209769 (+ 32690us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.221035 (+ 11266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1555184}
I20250629 01:51:47.221483 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 456) took 1674 ms. Trace:
I20250629 01:51:47.221577 1157 rpcz_store.cc:276] 0629 01:51:45.547012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.548562 (+ 1550us) service_pool.cc:225] Handling call
0629 01:51:47.221378 (+1672816us) ts_tablet_manager.cc:1145] Deleting tablet 196f1bd233ab4399a9d22a55b8bc2635
0629 01:51:47.221386 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.221458 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669358}
I20250629 01:51:47.221863 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 459) took 1674 ms. Trace:
I20250629 01:51:47.221958 1157 rpcz_store.cc:276] 0629 01:51:45.547482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.548086 (+ 604us) service_pool.cc:225] Handling call
0629 01:51:47.221751 (+1673665us) ts_tablet_manager.cc:1145] Deleting tablet 819ed44d98f14075b9a67b88dc44e86e
0629 01:51:47.221759 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.221837 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1658470}
I20250629 01:51:47.222265 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 460) took 1658 ms. Trace:
I20250629 01:51:47.222370 1157 rpcz_store.cc:276] 0629 01:51:45.563707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.563790 (+ 83us) service_pool.cc:225] Handling call
0629 01:51:47.222161 (+1658371us) ts_tablet_manager.cc:1145] Deleting tablet dd709b557b334434a6c2c4a60a996342
0629 01:51:47.222168 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.222237 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646812}
I20250629 01:51:47.222654 1157 tablet_replica.cc:331] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:47.222836 1157 raft_consensus.cc:2241] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.223009 1157 raft_consensus.cc:2270] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.223369 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_278cd76ddcb64217983536c8069d1803" } new_table_name: "default.Table_2c7fd136d3344a2f86e67efda6da2803" modify_external_catalogs: true
I20250629 01:51:47.223477 1157 ts_tablet_manager.cc:1905] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.224256 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet dfee69f9484046d8abe10d2f07033e09 (table default.Table_2bd4de753ed04021b2e3122b263bafab [id=40b322522a664b8f8bb8f39388b8ddf6]) successfully deleted
W20250629 01:51:47.224576 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 196f1bd233ab4399a9d22a55b8bc2635 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 196f1bd233ab4399a9d22a55b8bc2635 already in progress: creating tablet
W20250629 01:51:47.224793 986 leader_election.cc:343] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:47.224889 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 819ed44d98f14075b9a67b88dc44e86e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 819ed44d98f14075b9a67b88dc44e86e already in progress: creating tablet
W20250629 01:51:47.225208 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet dd709b557b334434a6c2c4a60a996342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd709b557b334434a6c2c4a60a996342 already in progress: creating tablet
W20250629 01:51:47.225487 850 leader_election.cc:343] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 5f60c48c924d49a186a8c81d154bdc54
I20250629 01:51:47.229055 1155 tablet_bootstrap.cc:492] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.229295 1155 ts_tablet_manager.cc:1397] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20250629 01:51:47.230187 1155 raft_consensus.cc:357] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.230331 1154 ts_tablet_manager.cc:1428] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250629 01:51:47.230441 1155 raft_consensus.cc:383] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.230543 1155 raft_consensus.cc:738] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.230600 1154 tablet_bootstrap.cc:492] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.230813 1155 consensus_queue.cc:260] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.231289 1155 ts_tablet_manager.cc:1428] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:47.231649 1155 tablet_bootstrap.cc:492] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.234706 1049 raft_consensus.cc:1273] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:47.235328 1694 consensus_queue.cc:1035] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:47.235787 1155 tablet_bootstrap.cc:654] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.236361 1158 tablet_replica.cc:331] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.236528 1158 raft_consensus.cc:2241] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.236670 1158 raft_consensus.cc:2270] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.238363 1158 ts_tablet_manager.cc:1905] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.238696 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e72b0bd01cbb4f8eaa5caecb2d75133a" } modify_external_catalogs: true
I20250629 01:51:47.238842 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e72b0bd01cbb4f8eaa5caecb2d75133a" } modify_external_catalogs: true
I20250629 01:51:47.239737 1154 tablet_bootstrap.cc:654] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.241040 850 leader_election.cc:304] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.241667 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e508393b9cb044c4b90194c8aab3184a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:47.241808 916 raft_consensus.cc:3058] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:47.242957 712 consensus_peers.cc:489] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:47.245007 916 raft_consensus.cc:2466] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:47.245645 988 leader_election.cc:304] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters: ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.246218 1364 raft_consensus.cc:2802] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
W20250629 01:51:47.249658 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 consensus state has no leader
I20250629 01:51:47.255035 1364 raft_consensus.cc:695] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:47.255324 1364 consensus_queue.cc:237] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.256177 1638 raft_consensus.cc:2747] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.263164 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_02ed750488084801a295ba2e0b78d76f" } new_table_name: "default.Table_860e06a9808a4f8d9be0cedf70de1be4" modify_external_catalogs: true
W20250629 01:51:47.236753 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e4d6f1eb43b24fda94aa0afcf4501195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4d6f1eb43b24fda94aa0afcf4501195 already in progress: creating tablet
I20250629 01:51:47.271227 1158 ts_tablet_manager.cc:1918] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.271320 1158 log.cc:1199] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d663e7dc7a384bf6bd0cbfa9d9e5e9f0
I20250629 01:51:47.271603 1158 ts_tablet_manager.cc:1939] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.272786 1155 tablet_bootstrap.cc:492] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.272857 1154 tablet_bootstrap.cc:492] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.272905 1155 ts_tablet_manager.cc:1397] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250629 01:51:47.273027 1154 ts_tablet_manager.cc:1397] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250629 01:51:47.273581 1155 raft_consensus.cc:357] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.273599 1154 raft_consensus.cc:357] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.273730 1155 raft_consensus.cc:383] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.273765 1154 raft_consensus.cc:383] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.273803 1155 raft_consensus.cc:738] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.273824 1154 raft_consensus.cc:738] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.273988 1155 consensus_queue.cc:260] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.274029 1154 consensus_queue.cc:260] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.274524 1155 ts_tablet_manager.cc:1428] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:47.274765 1155 tablet_bootstrap.cc:492] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.276211 1154 ts_tablet_manager.cc:1428] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:47.276448 1154 tablet_bootstrap.cc:492] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.276721 1155 tablet_bootstrap.cc:654] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.277779 1154 tablet_bootstrap.cc:654] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.278966 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 (table default.Table_737eb1cf8b184a14b253c25d114a14b7 [id=eda52aa556274b26af9896e638b51e7a]) successfully deleted
W20250629 01:51:47.287147 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f9f59fbe9634ac8b27f990d9496f637 consensus state has no leader
I20250629 01:51:47.290030 446 master_service.cc:948] ReplaceTablet: received request to replace tablet e3b17172ab8b4e0dad36ebaac776dc03 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.290405 455 master_service.cc:948] ReplaceTablet: received request to replace tablet ec0ca04e5f6b42df8567deb53a3e5f53 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.296706 1157 ts_tablet_manager.cc:1918] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.296834 1157 log.cc:1199] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/79e61d9576f8414ebf0074d57cdfb1f5
I20250629 01:51:47.297155 1157 ts_tablet_manager.cc:1939] T 79e61d9576f8414ebf0074d57cdfb1f5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:47.297825 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 457) took 1750 ms. Trace:
I20250629 01:51:47.297924 1157 rpcz_store.cc:276] 0629 01:51:45.547265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.592947 (+ 45682us) service_pool.cc:225] Handling call
0629 01:51:47.222605 (+1629658us) ts_tablet_manager.cc:1145] Deleting tablet 79e61d9576f8414ebf0074d57cdfb1f5
0629 01:51:47.222613 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.275030 (+ 52417us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.296614 (+ 21584us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.297776 (+ 1162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601187}
I20250629 01:51:47.298286 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 79e61d9576f8414ebf0074d57cdfb1f5 (table default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]) successfully deleted
I20250629 01:51:47.314280 1682 tablet.cc:1722] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250629 01:51:47.315790 1159 ts_tablet_manager.cc:1428] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.329s user 0.011s sys 0.008s
I20250629 01:51:47.318573 1681 tablet.cc:1722] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250629 01:51:47.319000 1159 tablet_bootstrap.cc:492] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:47.321702 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f7340594bca94873b0da94b3e42d46e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.321949 1159 tablet_bootstrap.cc:654] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:47.322201 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7340594bca94873b0da94b3e42d46e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.327055 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d7a4536b4ffa4887a2b30a00f22275c6" } new_table_name: "default.Table_e36a71de95204b6fbf931a4ce70fb501" modify_external_catalogs: true
I20250629 01:51:47.330498 1157 tablet_replica.cc:331] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:47.330672 1157 raft_consensus.cc:2241] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.330842 1157 raft_consensus.cc:2270] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.331277 1157 ts_tablet_manager.cc:1905] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.337859 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 241a5c9aee3f4297b67b18380ef5ef88 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.339001 1159 tablet_bootstrap.cc:492] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:47.339165 1159 ts_tablet_manager.cc:1397] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
W20250629 01:51:47.339639 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2045012c80d049eaabf7dec1df28b087 consensus state has no leader
I20250629 01:51:47.340916 1159 raft_consensus.cc:357] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.341063 1159 raft_consensus.cc:383] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.341136 1159 raft_consensus.cc:738] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.341329 1159 consensus_queue.cc:260] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.343803 451 master_service.cc:948] ReplaceTablet: received request to replace tablet ef3060d46ac24330be49103920dc1dee from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.345062 1159 ts_tablet_manager.cc:1428] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:51:47.347182 1159 tablet_bootstrap.cc:492] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:47.349189 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
I20250629 01:51:47.349804 1159 tablet_bootstrap.cc:654] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.352702 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_35d223e2a4d24c85a2a3780651c205e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.352878 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_33cae964f7ab41198ea114d25de7742f" } new_table_name: "default.Table_ca2bcac172414e6c9f264cb2202a76fa" modify_external_catalogs: true
W20250629 01:51:47.353215 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d223e2a4d24c85a2a3780651c205e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.355466 741 tablet_service.cc:1468] Processing CreateTablet for tablet 28b171a57f5b4bfd95bb1324a5ee982c (DEFAULT_TABLE table=default.Table_e33e44d3bb7d4104866f13a3bd938c15 [id=893ab7e2c42f490b8ec1e10450337b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.355891 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b171a57f5b4bfd95bb1324a5ee982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.356173 447 catalog_manager.cc:5582] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:47.356612 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
I20250629 01:51:47.356819 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_f07aa6eb42b6478781eb5a65dea53212 [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 6 got 0
I20250629 01:51:47.363613 1159 tablet_bootstrap.cc:492] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:47.363746 1159 ts_tablet_manager.cc:1397] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250629 01:51:47.363994 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d
I20250629 01:51:47.364415 1159 raft_consensus.cc:357] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.364570 1159 raft_consensus.cc:383] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.364639 1159 raft_consensus.cc:738] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.364773 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7d330f73937f409590ae615703808e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.364912 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.364862 1159 consensus_queue.cc:260] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:47.365314 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d330f73937f409590ae615703808e30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.366983 1159 ts_tablet_manager.cc:1428] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:47.367933 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.368166 1159 tablet_bootstrap.cc:492] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:47.368317 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.368999 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e3b17172ab8b4e0dad36ebaac776dc03
I20250629 01:51:47.369454 1159 tablet_bootstrap.cc:654] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.369495 446 catalog_manager.cc:6652] ReplaceTablet: tablet e3b17172ab8b4e0dad36ebaac776dc03 deleted and replaced by tablet 35a62c3fe4fb4220838deece8e46d074
I20250629 01:51:47.369851 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5149d5ed90ca4d6ca10f30e1694a0e1d" } modify_external_catalogs: true
I20250629 01:51:47.369899 884 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 with delete_type TABLET_DATA_DELETED (replaced by tablet 35a62c3fe4fb4220838deece8e46d074) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.370023 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5149d5ed90ca4d6ca10f30e1694a0e1d" } modify_external_catalogs: true
W20250629 01:51:47.371809 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d already in progress: creating tablet
I20250629 01:51:47.371857 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 with delete_type TABLET_DATA_DELETED (replaced by tablet 35a62c3fe4fb4220838deece8e46d074) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:47.372053 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d already in progress: creating tablet
W20250629 01:51:47.372501 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e3b17172ab8b4e0dad36ebaac776dc03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b17172ab8b4e0dad36ebaac776dc03 already in progress: creating tablet
I20250629 01:51:47.377338 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ec0ca04e5f6b42df8567deb53a3e5f53
I20250629 01:51:47.377671 1157 ts_tablet_manager.cc:1918] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.377768 1157 log.cc:1199] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/299922fa5e694bdf96f29f1522068656
I20250629 01:51:47.377962 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_657df5e298e548eab160b135980fba3a" } new_table_name: "default.Table_4c419cdd9add4db7b041836405c4856b" modify_external_catalogs: true
I20250629 01:51:47.378078 1157 ts_tablet_manager.cc:1939] T 299922fa5e694bdf96f29f1522068656 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:47.378134 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.378337 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 471) took 1653 ms. Trace:
I20250629 01:51:47.378427 1157 rpcz_store.cc:276] 0629 01:51:45.725303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.729366 (+ 4063us) service_pool.cc:225] Handling call
0629 01:51:47.330432 (+1601066us) ts_tablet_manager.cc:1145] Deleting tablet 299922fa5e694bdf96f29f1522068656
0629 01:51:47.330441 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.333974 (+ 3533us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.377567 (+ 43593us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.378309 (+ 742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577490}
I20250629 01:51:47.378783 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 299922fa5e694bdf96f29f1522068656 (table default.Table_ea5d8228a5e4443cb41d5c9baa8a3d27 [id=28e5371a950e42f8803d96b0ffa26cf8]) successfully deleted
I20250629 01:51:47.378890 884 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:47.378778 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ec0ca04e5f6b42df8567deb53a3e5f53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0ca04e5f6b42df8567deb53a3e5f53 already in progress: creating tablet
I20250629 01:51:47.379134 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.379282 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_82d6223100ea4aba8941923226f2baa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.379747 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ec0ca04e5f6b42df8567deb53a3e5f53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0ca04e5f6b42df8567deb53a3e5f53 already in progress: creating tablet
W20250629 01:51:47.381306 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82d6223100ea4aba8941923226f2baa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:47.381776 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0323947b135a4308903b1eed1ca4db71 consensus state has no leader
I20250629 01:51:47.382634 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 473) took 1632 ms. Trace:
I20250629 01:51:47.382738 1157 rpcz_store.cc:276] 0629 01:51:45.749857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.749909 (+ 52us) service_pool.cc:225] Handling call
0629 01:51:47.382519 (+1632610us) ts_tablet_manager.cc:1145] Deleting tablet df66984acfa44a99bf69096aff6d49ab
0629 01:51:47.382527 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.382607 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618689}
W20250629 01:51:47.383773 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet df66984acfa44a99bf69096aff6d49ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df66984acfa44a99bf69096aff6d49ab already in progress: creating tablet
I20250629 01:51:47.385253 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 474) took 1622 ms. Trace:
I20250629 01:51:47.385360 1157 rpcz_store.cc:276] 0629 01:51:45.763094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.764065 (+ 971us) service_pool.cc:225] Handling call
0629 01:51:47.385150 (+1621085us) ts_tablet_manager.cc:1145] Deleting tablet 8295e8178eff4d5eb95ac885092731e0
0629 01:51:47.385158 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.385229 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1592977}
W20250629 01:51:47.385720 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8295e8178eff4d5eb95ac885092731e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8295e8178eff4d5eb95ac885092731e0 already in progress: creating tablet
I20250629 01:51:47.387355 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 472) took 1641 ms. Trace:
I20250629 01:51:47.387457 1157 rpcz_store.cc:276] 0629 01:51:45.746289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.746779 (+ 490us) service_pool.cc:225] Handling call
0629 01:51:47.387243 (+1640464us) ts_tablet_manager.cc:1145] Deleting tablet 49333302a27f4ac3b76a1e492b3c1b60
0629 01:51:47.387251 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.387329 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1584655}
W20250629 01:51:47.387810 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 49333302a27f4ac3b76a1e492b3c1b60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49333302a27f4ac3b76a1e492b3c1b60 already in progress: creating tablet
I20250629 01:51:47.388957 741 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 with delete_type TABLET_DATA_DELETED (replaced by tablet 35a62c3fe4fb4220838deece8e46d074) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:47.389451 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e3b17172ab8b4e0dad36ebaac776dc03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b17172ab8b4e0dad36ebaac776dc03 already in progress: creating tablet
I20250629 01:51:47.390758 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 482) took 1497 ms. Trace:
I20250629 01:51:47.390859 1157 rpcz_store.cc:276] 0629 01:51:45.893596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.893650 (+ 54us) service_pool.cc:225] Handling call
0629 01:51:47.390661 (+1497011us) ts_tablet_manager.cc:1145] Deleting tablet 683c87ea2ad44cecb2366d4a846d9195
0629 01:51:47.390669 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.390733 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494739}
W20250629 01:51:47.391256 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 683c87ea2ad44cecb2366d4a846d9195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 683c87ea2ad44cecb2366d4a846d9195 already in progress: creating tablet
I20250629 01:51:47.394481 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 486) took 1486 ms. Trace:
I20250629 01:51:47.394618 1157 rpcz_store.cc:276] 0629 01:51:45.908357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:45.908422 (+ 65us) service_pool.cc:225] Handling call
0629 01:51:47.394351 (+1485929us) ts_tablet_manager.cc:1145] Deleting tablet fbcae02d893f4795a12d26af57b18b85
0629 01:51:47.394359 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.394445 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484234}
W20250629 01:51:47.394991 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet fbcae02d893f4795a12d26af57b18b85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbcae02d893f4795a12d26af57b18b85 already in progress: creating tablet
I20250629 01:51:47.396283 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 496) took 1242 ms. Trace:
I20250629 01:51:47.396384 1157 rpcz_store.cc:276] 0629 01:51:46.153747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.161610 (+ 7863us) service_pool.cc:225] Handling call
0629 01:51:47.396166 (+1234556us) ts_tablet_manager.cc:1145] Deleting tablet 2f7c1f1af39d47c191d34dc026f7f687
0629 01:51:47.396172 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.396257 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234410}
W20250629 01:51:47.396718 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2f7c1f1af39d47c191d34dc026f7f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7c1f1af39d47c191d34dc026f7f687 already in progress: creating tablet
I20250629 01:51:47.396958 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 28b171a57f5b4bfd95bb1324a5ee982c (DEFAULT_TABLE table=default.Table_e33e44d3bb7d4104866f13a3bd938c15 [id=893ab7e2c42f490b8ec1e10450337b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.397313 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b171a57f5b4bfd95bb1324a5ee982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.397564 884 tablet_service.cc:1468] Processing CreateTablet for tablet 28b171a57f5b4bfd95bb1324a5ee982c (DEFAULT_TABLE table=default.Table_e33e44d3bb7d4104866f13a3bd938c15 [id=893ab7e2c42f490b8ec1e10450337b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.397891 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28b171a57f5b4bfd95bb1324a5ee982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.398103 1155 tablet_bootstrap.cc:492] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.398244 1155 ts_tablet_manager.cc:1397] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.000s
I20250629 01:51:47.398866 1155 raft_consensus.cc:357] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.398940 741 tablet_service.cc:1468] Processing CreateTablet for tablet 959837d39a9a4a6f8735a906b3a53222 (DEFAULT_TABLE table=default.Table_57c0327ef1554e008fc2b4cbc62b28f9 [id=66f9a4f451aa415fa8e5f94682b04a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.399032 1155 raft_consensus.cc:383] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.399101 1155 raft_consensus.cc:738] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.399271 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959837d39a9a4a6f8735a906b3a53222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.399333 1155 consensus_queue.cc:260] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.402356 1155 ts_tablet_manager.cc:1428] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:47.402519 1364 raft_consensus.cc:491] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.402638 1364 raft_consensus.cc:513] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.403177 1364 leader_election.cc:290] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.403676 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa48f94ac0e48e39578a6e8af5c8e9c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:47.403856 916 raft_consensus.cc:2466] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:47.403918 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa48f94ac0e48e39578a6e8af5c8e9c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.404109 775 raft_consensus.cc:2466] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:47.404382 988 leader_election.cc:304] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:47.404428 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 497) took 1242 ms. Trace:
I20250629 01:51:47.404513 1157 rpcz_store.cc:276] 0629 01:51:46.161903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.161945 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:47.404321 (+1242376us) ts_tablet_manager.cc:1145] Deleting tablet a90b206d58d04acdb1a8cb0c45d68c31
0629 01:51:47.404329 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.404401 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241293}
I20250629 01:51:47.404650 1155 tablet_bootstrap.cc:492] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:47.404822 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a90b206d58d04acdb1a8cb0c45d68c31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a90b206d58d04acdb1a8cb0c45d68c31 already in progress: creating tablet
I20250629 01:51:47.405050 1364 raft_consensus.cc:2802] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:47.405144 1364 raft_consensus.cc:491] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:47.405218 1364 raft_consensus.cc:3058] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.406044 1155 tablet_bootstrap.cc:654] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.407593 1364 raft_consensus.cc:513] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.407796 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7fed58fcd48d4bd6ba7e1ce4d4c40635"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.408219 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fed58fcd48d4bd6ba7e1ce4d4c40635 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.408361 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa48f94ac0e48e39578a6e8af5c8e9c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:47.408504 916 raft_consensus.cc:3058] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.408519 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 959837d39a9a4a6f8735a906b3a53222 (DEFAULT_TABLE table=default.Table_57c0327ef1554e008fc2b4cbc62b28f9 [id=66f9a4f451aa415fa8e5f94682b04a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.408834 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959837d39a9a4a6f8735a906b3a53222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.409201 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d01934d1cb70450dbb1b6cf0c40c2129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.409741 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d01934d1cb70450dbb1b6cf0c40c2129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.410645 916 raft_consensus.cc:2466] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:47.411649 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ef3060d46ac24330be49103920dc1dee
I20250629 01:51:47.412487 451 catalog_manager.cc:6652] ReplaceTablet: tablet ef3060d46ac24330be49103920dc1dee deleted and replaced by tablet 86099eb9e40a4ea389afc059f90a97ef
I20250629 01:51:47.413009 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet ef3060d46ac24330be49103920dc1dee with delete_type TABLET_DATA_DELETED (replaced by tablet 86099eb9e40a4ea389afc059f90a97ef) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.413007 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ef3060d46ac24330be49103920dc1dee with delete_type TABLET_DATA_DELETED (replaced by tablet 86099eb9e40a4ea389afc059f90a97ef) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:47.413568 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ef3060d46ac24330be49103920dc1dee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3060d46ac24330be49103920dc1dee already in progress: creating tablet
I20250629 01:51:47.413661 988 leader_election.cc:304] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
W20250629 01:51:47.413779 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ef3060d46ac24330be49103920dc1dee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3060d46ac24330be49103920dc1dee already in progress: creating tablet
I20250629 01:51:47.413875 1710 raft_consensus.cc:2802] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:47.415248 884 tablet_service.cc:1515] Processing DeleteTablet for tablet ef3060d46ac24330be49103920dc1dee with delete_type TABLET_DATA_DELETED (replaced by tablet 86099eb9e40a4ea389afc059f90a97ef) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.415390 1710 raft_consensus.cc:695] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:47.415524 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 498) took 1252 ms. Trace:
I20250629 01:51:47.415645 1710 consensus_queue.cc:237] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.415616 1157 rpcz_store.cc:276] 0629 01:51:46.162214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.162246 (+ 32us) service_pool.cc:225] Handling call
0629 01:51:47.415065 (+1252819us) ts_tablet_manager.cc:1145] Deleting tablet f6d79ac3c5f04b5db295c93c3aae92cb
0629 01:51:47.415071 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.415138 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252694}
W20250629 01:51:47.416603 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f6d79ac3c5f04b5db295c93c3aae92cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6d79ac3c5f04b5db295c93c3aae92cb already in progress: creating tablet
I20250629 01:51:47.416851 884 tablet_service.cc:1468] Processing CreateTablet for tablet 959837d39a9a4a6f8735a906b3a53222 (DEFAULT_TABLE table=default.Table_57c0327ef1554e008fc2b4cbc62b28f9 [id=66f9a4f451aa415fa8e5f94682b04a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.417193 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 959837d39a9a4a6f8735a906b3a53222. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.417465 751 tablet_service.cc:1468] Processing CreateTablet for tablet 996ad157d7804856bdb9b606bd412710 (DEFAULT_TABLE table=default.Table_87ba700594ec433ebf0252c79cdbdab9 [id=ed5bc9dcb04a4701b2c368255c98ab07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.417771 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 996ad157d7804856bdb9b606bd412710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.421137 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bfa48f94ac0e48e39578a6e8af5c8e9c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:47.421280 775 raft_consensus.cc:3058] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.416251 1159 tablet_bootstrap.cc:492] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:47.423251 1159 ts_tablet_manager.cc:1397] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250629 01:51:47.423928 1159 raft_consensus.cc:357] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.424072 1159 raft_consensus.cc:383] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.424171 1159 raft_consensus.cc:738] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.424038 775 raft_consensus.cc:2466] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:47.424376 1159 consensus_queue.cc:260] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.424657 1364 leader_election.cc:290] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.425514 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ed9b0a279de49078207546c6ddbc21f" } modify_external_catalogs: true
I20250629 01:51:47.425664 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ed9b0a279de49078207546c6ddbc21f" } modify_external_catalogs: true
I20250629 01:51:47.428553 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.428635 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d471323a04d4c0389ad1d253aa5cd36" } modify_external_catalogs: true
I20250629 01:51:47.428761 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d471323a04d4c0389ad1d253aa5cd36" } modify_external_catalogs: true
W20250629 01:51:47.429072 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 because the tablet was not found. No further retry: Not found: Tablet not found: d663e7dc7a384bf6bd0cbfa9d9e5e9f0
I20250629 01:51:47.430040 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 996ad157d7804856bdb9b606bd412710 (DEFAULT_TABLE table=default.Table_87ba700594ec433ebf0252c79cdbdab9 [id=ed5bc9dcb04a4701b2c368255c98ab07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.430379 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 996ad157d7804856bdb9b606bd412710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.430756 883 tablet_service.cc:1468] Processing CreateTablet for tablet dbfcb9233ace429db5daa593c2a1dcbe (DEFAULT_TABLE table=default.Table_b50cf2cfa18641428019543c0982188d [id=f05487296c954ad08343c2a49b1934ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.431067 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfcb9233ace429db5daa593c2a1dcbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.431578 450 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_6316ef46c0a6445190d1fa69105eacaa [id=95ccd48727404118a0e2c26dd2103a12]). Expected version 1 got 0
I20250629 01:51:47.434121 739 tablet_service.cc:1468] Processing CreateTablet for tablet ca13eb6bdcd044ec948ea699857a30f8 (DEFAULT_TABLE table=default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 [id=2e90d163f6834cf79730e835b629994a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.434441 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca13eb6bdcd044ec948ea699857a30f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.434597 1019 tablet_service.cc:1468] Processing CreateTablet for tablet dbfcb9233ace429db5daa593c2a1dcbe (DEFAULT_TABLE table=default.Table_b50cf2cfa18641428019543c0982188d [id=f05487296c954ad08343c2a49b1934ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.434877 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfcb9233ace429db5daa593c2a1dcbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.437057 886 tablet_service.cc:1515] Processing DeleteTablet for tablet f1e0de66406e4ce89ce56fa447fe9895 with delete_type TABLET_DATA_DELETED (replaced by tablet be8e0946a20340b38835c4d8daeaa7b8) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.437500 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5a1604552122400c8fb0aca541834525 (DEFAULT_TABLE table=default.Table_0a1d504b9fd444a38d8e74bd4b2efd44 [id=2a312e0d477041258d09ef05a48b373b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.437772 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1604552122400c8fb0aca541834525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.439725 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 56b9ef027c9d42048e4a6dff221df793
I20250629 01:51:47.439996 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 499) took 1277 ms. Trace:
I20250629 01:51:47.440018 1018 tablet_service.cc:1468] Processing CreateTablet for tablet ca13eb6bdcd044ec948ea699857a30f8 (DEFAULT_TABLE table=default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 [id=2e90d163f6834cf79730e835b629994a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.440338 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca13eb6bdcd044ec948ea699857a30f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.440380 745 tablet_service.cc:1468] Processing CreateTablet for tablet 5a1604552122400c8fb0aca541834525 (DEFAULT_TABLE table=default.Table_0a1d504b9fd444a38d8e74bd4b2efd44 [id=2a312e0d477041258d09ef05a48b373b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.440662 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1604552122400c8fb0aca541834525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.440987 1157 rpcz_store.cc:276] 0629 01:51:46.162466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.162497 (+ 31us) service_pool.cc:225] Handling call
0629 01:51:47.439865 (+1277368us) ts_tablet_manager.cc:1145] Deleting tablet 7cc18d1154ef4d05b3b82180a9c1ca8b
0629 01:51:47.439872 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.439957 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1277242}
I20250629 01:51:47.441336 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 500) took 1278 ms. Trace:
I20250629 01:51:47.441428 1157 rpcz_store.cc:276] 0629 01:51:46.162715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.162746 (+ 31us) service_pool.cc:225] Handling call
0629 01:51:47.441224 (+1278478us) ts_tablet_manager.cc:1145] Deleting tablet 2845d8fbf3f34c769027b9b6bb3e1f08
0629 01:51:47.441232 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.441301 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278340}
I20250629 01:51:47.441885 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 503) took 1188 ms. Trace:
I20250629 01:51:47.441977 1157 rpcz_store.cc:276] 0629 01:51:46.253900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.254063 (+ 163us) service_pool.cc:225] Handling call
0629 01:51:47.441596 (+1187533us) ts_tablet_manager.cc:1145] Deleting tablet 9fe9674ffeb44ba6ab60de59d4c08670
0629 01:51:47.441603 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.441856 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1187391}
I20250629 01:51:47.442243 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 504) took 1188 ms. Trace:
I20250629 01:51:47.442322 1157 rpcz_store.cc:276] 0629 01:51:46.254023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.257277 (+ 3254us) service_pool.cc:225] Handling call
0629 01:51:47.442137 (+1184860us) ts_tablet_manager.cc:1145] Deleting tablet 07e0727102d54294a7c13a4aeef311e1
0629 01:51:47.442144 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.442216 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184688}
I20250629 01:51:47.442562 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 505) took 1177 ms. Trace:
I20250629 01:51:47.442647 1157 rpcz_store.cc:276] 0629 01:51:46.265373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.268160 (+ 2787us) service_pool.cc:225] Handling call
0629 01:51:47.442470 (+1174310us) ts_tablet_manager.cc:1145] Deleting tablet e3b94f8250a143b096703f2b0386e3cd
0629 01:51:47.442477 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.442532 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1174161}
I20250629 01:51:47.442857 1157 tablet_replica.cc:331] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:47.443042 1157 raft_consensus.cc:2241] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.443188 1157 raft_consensus.cc:2270] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.443441 883 tablet_service.cc:1468] Processing CreateTablet for tablet ca13eb6bdcd044ec948ea699857a30f8 (DEFAULT_TABLE table=default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 [id=2e90d163f6834cf79730e835b629994a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.443655 1157 ts_tablet_manager.cc:1905] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.443713 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca13eb6bdcd044ec948ea699857a30f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.443928 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 5a1604552122400c8fb0aca541834525 (DEFAULT_TABLE table=default.Table_0a1d504b9fd444a38d8e74bd4b2efd44 [id=2a312e0d477041258d09ef05a48b373b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.444232 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1604552122400c8fb0aca541834525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.444335 744 tablet_service.cc:1468] Processing CreateTablet for tablet 49844d26db6447de8c323376c7756437 (DEFAULT_TABLE table=default.Table_93dfa58f836843938471c38606d8e6b0 [id=9584ca8a4c9c4ee3b50fec49318fa9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.444605 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49844d26db6447de8c323376c7756437. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.447985 886 tablet_service.cc:1468] Processing CreateTablet for tablet 49844d26db6447de8c323376c7756437 (DEFAULT_TABLE table=default.Table_93dfa58f836843938471c38606d8e6b0 [id=9584ca8a4c9c4ee3b50fec49318fa9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.448320 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49844d26db6447de8c323376c7756437. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.448709 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 49844d26db6447de8c323376c7756437 (DEFAULT_TABLE table=default.Table_93dfa58f836843938471c38606d8e6b0 [id=9584ca8a4c9c4ee3b50fec49318fa9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.449188 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49844d26db6447de8c323376c7756437. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.450922 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_711c32a531fd46489436ed1a683d3297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.451352 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_711c32a531fd46489436ed1a683d3297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.452375 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6864f50c0e624776b4090fb5368ae3dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.452490 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cc18d1154ef4d05b3b82180a9c1ca8b already in progress: creating tablet
W20250629 01:51:47.452683 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2845d8fbf3f34c769027b9b6bb3e1f08 already in progress: creating tablet
W20250629 01:51:47.452831 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6864f50c0e624776b4090fb5368ae3dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:47.452883 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 07e0727102d54294a7c13a4aeef311e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07e0727102d54294a7c13a4aeef311e1 already in progress: creating tablet
W20250629 01:51:47.453060 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e3b94f8250a143b096703f2b0386e3cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b94f8250a143b096703f2b0386e3cd already in progress: creating tablet
I20250629 01:51:47.453514 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b9ef027c9d42048e4a6dff221df793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.453917 1159 ts_tablet_manager.cc:1428] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20250629 01:51:47.454535 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cca695fad03947debe4d1d515b3c73d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.454903 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cca695fad03947debe4d1d515b3c73d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.457011 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b9ef027c9d42048e4a6dff221df793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.457289 1159 tablet_bootstrap.cc:492] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:47.457552 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 56b9ef027c9d42048e4a6dff221df793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56b9ef027c9d42048e4a6dff221df793 already in progress: creating tablet
I20250629 01:51:47.459086 884 tablet_service.cc:1468] Processing CreateTablet for tablet 996ad157d7804856bdb9b606bd412710 (DEFAULT_TABLE table=default.Table_87ba700594ec433ebf0252c79cdbdab9 [id=ed5bc9dcb04a4701b2c368255c98ab07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.459388 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 996ad157d7804856bdb9b606bd412710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.460341 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b9ef027c9d42048e4a6dff221df793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.460839 1155 tablet_bootstrap.cc:492] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
W20250629 01:51:47.460877 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 56b9ef027c9d42048e4a6dff221df793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56b9ef027c9d42048e4a6dff221df793 already in progress: creating tablet
I20250629 01:51:47.461053 1155 ts_tablet_manager.cc:1397] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20250629 01:51:47.461794 1155 raft_consensus.cc:357] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.461998 1155 raft_consensus.cc:383] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.462116 1155 raft_consensus.cc:738] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.462348 1155 consensus_queue.cc:260] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.463956 1155 ts_tablet_manager.cc:1428] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:47.464255 1155 tablet_bootstrap.cc:492] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.465592 1155 tablet_bootstrap.cc:654] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.468025 1159 tablet_bootstrap.cc:654] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.474647 751 tablet_service.cc:1468] Processing CreateTablet for tablet dbfcb9233ace429db5daa593c2a1dcbe (DEFAULT_TABLE table=default.Table_b50cf2cfa18641428019543c0982188d [id=f05487296c954ad08343c2a49b1934ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.474965 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbfcb9233ace429db5daa593c2a1dcbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:47.475663 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5d45854191247bf8172d0bbfd01b6db consensus state has no leader
I20250629 01:51:47.476763 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_6316ef46c0a6445190d1fa69105eacaa [id=95ccd48727404118a0e2c26dd2103a12]). Expected version 1 got 0
I20250629 01:51:47.477620 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_da7726c4aa4a4bf893ec5b19264cd6ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.478070 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da7726c4aa4a4bf893ec5b19264cd6ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.483268 1154 tablet_bootstrap.cc:492] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.483399 1154 ts_tablet_manager.cc:1397] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.207s user 0.018s sys 0.071s
I20250629 01:51:47.484077 1154 raft_consensus.cc:357] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.484261 1154 raft_consensus.cc:383] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.484345 1154 raft_consensus.cc:738] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.484730 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ebbaf6a5d4294ece9bc2e803e056a2e7" } new_table_name: "default.Table_fb0c134a9a1548cd8cebc3508bde937a" modify_external_catalogs: true
I20250629 01:51:47.485934 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 56b60431b72e4792aa1c32fdee40cdb9 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.490360 1154 consensus_queue.cc:260] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.490808 1154 ts_tablet_manager.cc:1428] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250629 01:51:47.491071 1154 tablet_bootstrap.cc:492] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.497550 453 catalog_manager.cc:5582] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:47.497864 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_6316ef46c0a6445190d1fa69105eacaa [id=95ccd48727404118a0e2c26dd2103a12]). Expected version 1 got 0
I20250629 01:51:47.498499 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:51:47.499017 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6316ef46c0a6445190d1fa69105eacaa" } new_table_name: "default.Table_974936d8b1084dcc8e5befb81e238b37" modify_external_catalogs: true
I20250629 01:51:47.501305 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:47.502132 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8a880c9bb4b94d3b9b12f28d48565f06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a880c9bb4b94d3b9b12f28d48565f06 already in progress: creating tablet
I20250629 01:51:47.503592 1155 tablet_bootstrap.cc:492] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.503767 1155 ts_tablet_manager.cc:1397] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250629 01:51:47.504456 1155 raft_consensus.cc:357] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.504670 1155 raft_consensus.cc:383] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.504763 1155 raft_consensus.cc:738] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.505023 1155 consensus_queue.cc:260] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.506573 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6ca02bcd9467496c9057b3de9c1b1725
I20250629 01:51:47.507264 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.507409 1155 ts_tablet_manager.cc:1428] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:47.507658 1155 tablet_bootstrap.cc:492] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.508545 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:47.509120 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6ca02bcd9467496c9057b3de9c1b1725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ca02bcd9467496c9057b3de9c1b1725 already in progress: creating tablet
I20250629 01:51:47.509404 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_f07aa6eb42b6478781eb5a65dea53212 [id=9ff841a004c344c688f5abf6b59b5dce] alter complete (version 6)
I20250629 01:51:47.509968 1155 tablet_bootstrap.cc:654] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:47.510102 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6ca02bcd9467496c9057b3de9c1b1725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ca02bcd9467496c9057b3de9c1b1725 already in progress: creating tablet
I20250629 01:51:47.510463 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.511123 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:47.511592 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8a880c9bb4b94d3b9b12f28d48565f06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a880c9bb4b94d3b9b12f28d48565f06 already in progress: creating tablet
I20250629 01:51:47.511765 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.515427 1154 tablet_bootstrap.cc:654] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.524405 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f with delete_type TABLET_DATA_DELETED (replaced by tablet f898483ab9514132be8537ad03720097) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.526978 1157 ts_tablet_manager.cc:1918] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.527134 1157 log.cc:1199] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5f60c48c924d49a186a8c81d154bdc54
I20250629 01:51:47.527500 1157 ts_tablet_manager.cc:1939] T 5f60c48c924d49a186a8c81d154bdc54 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:47.528340 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 516) took 1171 ms. Trace:
I20250629 01:51:47.528446 1157 rpcz_store.cc:276] 0629 01:51:46.357035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.357396 (+ 361us) service_pool.cc:225] Handling call
0629 01:51:47.442805 (+1085409us) ts_tablet_manager.cc:1145] Deleting tablet 5f60c48c924d49a186a8c81d154bdc54
0629 01:51:47.442811 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.474208 (+ 31397us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.517160 (+ 42952us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.528281 (+ 11121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080516}
I20250629 01:51:47.529634 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 5f60c48c924d49a186a8c81d154bdc54 (table default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]) successfully deleted
I20250629 01:51:47.536724 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 518) took 1152 ms. Trace:
W20250629 01:51:47.537846 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6b6d5ff92c343cb8f1b3039b84390af consensus state has no leader
I20250629 01:51:47.541276 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6429e6545a1d4d5bb1b37b6799c105ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.541734 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6429e6545a1d4d5bb1b37b6799c105ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.542461 1157 rpcz_store.cc:276] 0629 01:51:46.384615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.384655 (+ 40us) service_pool.cc:225] Handling call
0629 01:51:47.536588 (+1151933us) ts_tablet_manager.cc:1145] Deleting tablet 1c995ce14c9a4966b842fcba520cf87d
0629 01:51:47.536596 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.536680 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":937267}
I20250629 01:51:47.543108 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_33ab0761369a4bd280345f65e285d02b" } new_table_name: "default.Table_1a2a249712224a0fa06b23867c043cfa" modify_external_catalogs: true
W20250629 01:51:47.542932 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1c995ce14c9a4966b842fcba520cf87d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c995ce14c9a4966b842fcba520cf87d already in progress: creating tablet
W20250629 01:51:47.545742 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 already in progress: creating tablet
W20250629 01:51:47.546444 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 already in progress: creating tablet
I20250629 01:51:47.549558 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.549803 1158 tablet_replica.cc:331] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.549999 1158 raft_consensus.cc:2241] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.550140 1158 raft_consensus.cc:2270] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.551371 1158 ts_tablet_manager.cc:1905] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.551541 773 raft_consensus.cc:1273] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:47.552453 916 raft_consensus.cc:1273] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:47.553167 1710 consensus_queue.cc:1035] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:47.554358 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7a640451934bff8672cd88dc86c529 with delete_type TABLET_DATA_DELETED (replaced by tablet d456410cd09845dabd979c123f70741b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.558372 1720 consensus_queue.cc:1035] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:47.561525 1158 ts_tablet_manager.cc:1918] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.562181 1158 log.cc:1199] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b7c7f2ac44734a39ab6434a74c91b9e4
I20250629 01:51:47.562716 1158 ts_tablet_manager.cc:1939] T b7c7f2ac44734a39ab6434a74c91b9e4 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.565212 1158 tablet_replica.cc:331] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.565683 1158 raft_consensus.cc:2241] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.565726 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b7c7f2ac44734a39ab6434a74c91b9e4 (table default.Table_fa24ba8423a34245b118894025a8b4f4 [id=a07cbcbb14984c538bdd7d1441d52055]) successfully deleted
I20250629 01:51:47.566129 1158 raft_consensus.cc:2270] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:47.566522 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d1a8a9bc691e4b158276f34a51c0dc61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1a8a9bc691e4b158276f34a51c0dc61 already in progress: creating tablet
W20250629 01:51:47.567080 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2ec92eddfabf40918a6cabaff5fe113a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ec92eddfabf40918a6cabaff5fe113a already in progress: creating tablet
I20250629 01:51:47.567121 1158 ts_tablet_manager.cc:1905] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.568698 1157 tablet_replica.cc:331] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:47.569423 1154 tablet_bootstrap.cc:492] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.569566 1154 ts_tablet_manager.cc:1397] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.001s
I20250629 01:51:47.570019 1159 tablet_bootstrap.cc:492] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:47.570161 1159 ts_tablet_manager.cc:1397] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.004s
I20250629 01:51:47.570206 1154 raft_consensus.cc:357] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.570375 1154 raft_consensus.cc:383] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.570439 1154 raft_consensus.cc:738] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.570669 1154 consensus_queue.cc:260] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.570796 1159 raft_consensus.cc:357] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.570955 1159 raft_consensus.cc:383] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.571018 1159 raft_consensus.cc:738] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.571115 1154 ts_tablet_manager.cc:1428] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:47.571427 1157 raft_consensus.cc:2241] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:47.571691 1157 pending_rounds.cc:70] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Trying to abort 1 pending ops.
I20250629 01:51:47.571745 1157 pending_rounds.cc:77] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Aborting op as it isn't in flight: id { term: 1 index: 4 } timestamp: 7172759169487560704 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "a72c6040421642578fac307ea9914ef6" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 4 new_table_name: "default.Table_eab06a5f9cbc4248a1757e3b5518545b" dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" new_extra_config { } }
W20250629 01:51:47.571944 1157 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:56376: Aborted: Op aborted
I20250629 01:51:47.572053 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:56376 (request call id 511) took 1261 ms. Trace:
I20250629 01:51:47.572173 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f6d8cf186fc4934ac5690bac0754373" } modify_external_catalogs: true
I20250629 01:51:47.572304 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f6d8cf186fc4934ac5690bac0754373" } modify_external_catalogs: true
I20250629 01:51:47.572140 1157 rpcz_store.cc:276] 0629 01:51:46.310615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.310659 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:47.572032 (+1261373us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0629 01:51:46.613135 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0629 01:51:46.613218 (+ 83us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0629 01:51:46.613225 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0629 01:51:46.613230 (+ 5us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0629 01:51:46.613262 (+ 32us) alter_schema_op.cc:128] START. Timestamp: 1751161906613174
0629 01:51:46.613264 (+ 2us) op_driver.cc:348] REPLICATION: starting
0629 01:51:46.613460 (+ 196us) log.cc:844] Serialized 173 byte log entry
0629 01:51:47.571882 (+958422us) op_driver.cc:464] REPLICATION: finished
0629 01:51:47.571890 (+ 8us) op_driver.cc:389] HandleFailure(Aborted: Op aborted)
0629 01:51:47.571894 (+ 4us) alter_schema_op.cc:171] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":302153,"prepare.run_cpu_time_us":9396,"prepare.run_wall_time_us":112605,"raft.queue_time_us":72,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":242,"replication_time_us":958590,"thread_start_us":414507,"threads_started":2,"wal-append.queue_time_us":133814}]]}
W20250629 01:51:47.573418 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ea681bd91e54e4898bcfb72bed6198f consensus state has no leader
I20250629 01:51:47.576826 1154 tablet_bootstrap.cc:492] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.576854 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.580771 1154 tablet_bootstrap.cc:654] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.580801 1159 consensus_queue.cc:260] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.581125 1157 raft_consensus.cc:2270] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.583436 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ecd89721aae04a9abed2f4ff44cfd3dd" } new_table_name: "default.Table_e998111141ca474dbff482e945a67465" modify_external_catalogs: true
I20250629 01:51:47.584558 1155 tablet_bootstrap.cc:492] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.584734 1155 ts_tablet_manager.cc:1397] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.077s user 0.005s sys 0.002s
I20250629 01:51:47.584795 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3821b9a8d745433382baadfd96b6c260"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.585238 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3821b9a8d745433382baadfd96b6c260 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.585467 1155 raft_consensus.cc:357] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.585608 1155 raft_consensus.cc:383] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.585690 1155 raft_consensus.cc:738] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.585894 1155 consensus_queue.cc:260] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.586297 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_307dc406b50143efa856b28a689b1fe1" } modify_external_catalogs: true
I20250629 01:51:47.586450 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_307dc406b50143efa856b28a689b1fe1" } modify_external_catalogs: true
I20250629 01:51:47.587041 1155 ts_tablet_manager.cc:1428] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:47.589057 1155 tablet_bootstrap.cc:492] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.590655 1155 tablet_bootstrap.cc:654] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.592239 1681 tablet.cc:1722] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:47.596319 1689 raft_consensus.cc:491] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.596447 1689 raft_consensus.cc:513] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.597057 1689 leader_election.cc:290] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:47.599063 850 leader_election.cc:343] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f1e0de66406e4ce89ce56fa447fe9895
I20250629 01:51:47.600713 1690 tablet.cc:1722] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:51:47.604555 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7908afa2abaa4521b44f954e0790f986 consensus state has no leader
I20250629 01:51:47.605672 1158 ts_tablet_manager.cc:1918] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:47.607262 850 leader_election.cc:343] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: f1e0de66406e4ce89ce56fa447fe9895
I20250629 01:51:47.607378 850 leader_election.cc:304] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.607698 1689 raft_consensus.cc:2747] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.602411 1682 tablet.cc:1722] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:47.609517 1158 log.cc:1199] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0b7a640451934bff8672cd88dc86c529
I20250629 01:51:47.609872 1158 ts_tablet_manager.cc:1939] T 0b7a640451934bff8672cd88dc86c529 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.610594 1158 tablet_replica.cc:331] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.610791 1158 raft_consensus.cc:2241] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.610932 1158 raft_consensus.cc:2270] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.611150 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0b7a640451934bff8672cd88dc86c529 (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]) successfully deleted
I20250629 01:51:47.628078 1155 tablet_bootstrap.cc:492] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.628477 1158 ts_tablet_manager.cc:1905] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.629817 1159 ts_tablet_manager.cc:1428] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.060s user 0.001s sys 0.001s
I20250629 01:51:47.634229 1157 ts_tablet_manager.cc:1905] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.636658 1159 tablet_bootstrap.cc:492] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:47.638818 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_974936d8b1084dcc8e5befb81e238b37 [id=95ccd48727404118a0e2c26dd2103a12]). Expected version 2 got 0
I20250629 01:51:47.642938 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d194cd6079254aa794545ca8273a3011 (DEFAULT_TABLE table=default.Table_f7340594bca94873b0da94b3e42d46e7 [id=67ef484c2dfa4643b9f1d7c2af465b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.643275 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d194cd6079254aa794545ca8273a3011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.644356 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 44ab886d05e0468ab0e7b156a32dd392 (DEFAULT_TABLE table=default.Table_d2e4a152a81f434eb4a24f2dd01af367 [id=35de30bc6f17483eb80fef1bf8ce6824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.644606 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ab886d05e0468ab0e7b156a32dd392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.648175 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 56b60431b72e4792aa1c32fdee40cdb9
I20250629 01:51:47.648313 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 1e4a173d9e70453c91d8ff79a710bc00 (DEFAULT_TABLE table=default.Table_53f9893ad8274a76ad3cea4fd31594c9 [id=6a23a4c8cd70412fa9957be6c583be80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.648526 454 catalog_manager.cc:6652] ReplaceTablet: tablet 56b60431b72e4792aa1c32fdee40cdb9 deleted and replaced by tablet d73c2eb0d69e467b94a250e1790d8142
I20250629 01:51:47.648597 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4a173d9e70453c91d8ff79a710bc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.649323 883 tablet_service.cc:1468] Processing CreateTablet for tablet 1e4a173d9e70453c91d8ff79a710bc00 (DEFAULT_TABLE table=default.Table_53f9893ad8274a76ad3cea4fd31594c9 [id=6a23a4c8cd70412fa9957be6c583be80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.649627 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4a173d9e70453c91d8ff79a710bc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.651067 886 tablet_service.cc:1468] Processing CreateTablet for tablet 44ab886d05e0468ab0e7b156a32dd392 (DEFAULT_TABLE table=default.Table_d2e4a152a81f434eb4a24f2dd01af367 [id=35de30bc6f17483eb80fef1bf8ce6824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.651386 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ab886d05e0468ab0e7b156a32dd392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.652304 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a40c1a8eec9549a8bdf695d6516440a8
I20250629 01:51:47.652313 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 2d37334db2b9465a8fcf5e849d5d3b0c (DEFAULT_TABLE table=default.Table_3827f711f3174f66b469302de8091583 [id=e11fdfa462d54c6c83b5582fd119a925]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.652575 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d37334db2b9465a8fcf5e849d5d3b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.654579 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ea4ec96134d342ad8386604b1305651d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.655016 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea4ec96134d342ad8386604b1305651d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.655705 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1a2a249712224a0fa06b23867c043cfa" } modify_external_catalogs: true
I20250629 01:51:47.655867 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1a2a249712224a0fa06b23867c043cfa" } modify_external_catalogs: true
I20250629 01:51:47.656138 884 tablet_service.cc:1468] Processing CreateTablet for tablet d194cd6079254aa794545ca8273a3011 (DEFAULT_TABLE table=default.Table_f7340594bca94873b0da94b3e42d46e7 [id=67ef484c2dfa4643b9f1d7c2af465b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.656419 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d194cd6079254aa794545ca8273a3011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.657212 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a692f00ba244a80b9a6d07029f8d27e from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.657994 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.658145 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 with delete_type TABLET_DATA_DELETED (replaced by tablet d73c2eb0d69e467b94a250e1790d8142) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.660085 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d7cbc2e6148846f9a58239b089e5a052
I20250629 01:51:47.660658 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_974936d8b1084dcc8e5befb81e238b37 [id=95ccd48727404118a0e2c26dd2103a12] alter complete (version 2)
I20250629 01:51:47.661060 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet d7cbc2e6148846f9a58239b089e5a052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.663534 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1ea681bd91e54e4898bcfb72bed6198f
I20250629 01:51:47.665004 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.665347 1155 ts_tablet_manager.cc:1397] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250629 01:51:47.666005 1154 tablet_bootstrap.cc:492] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.666133 1154 ts_tablet_manager.cc:1397] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.001s
I20250629 01:51:47.666831 1154 raft_consensus.cc:357] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.667035 1154 raft_consensus.cc:383] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.667114 1154 raft_consensus.cc:738] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.667302 1154 consensus_queue.cc:260] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.670830 1155 raft_consensus.cc:357] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.671141 1155 raft_consensus.cc:383] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.671234 1155 raft_consensus.cc:738] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.672662 744 tablet_service.cc:1468] Processing CreateTablet for tablet d194cd6079254aa794545ca8273a3011 (DEFAULT_TABLE table=default.Table_f7340594bca94873b0da94b3e42d46e7 [id=67ef484c2dfa4643b9f1d7c2af465b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.673501 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2d37334db2b9465a8fcf5e849d5d3b0c (DEFAULT_TABLE table=default.Table_3827f711f3174f66b469302de8091583 [id=e11fdfa462d54c6c83b5582fd119a925]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.674491 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d61687e19d994e87b09e55bd3b9476d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.674492 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.674513 1159 tablet_bootstrap.cc:654] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.674938 1154 ts_tablet_manager.cc:1428] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:47.674875 1155 consensus_queue.cc:260] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.675293 1155 ts_tablet_manager.cc:1428] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20250629 01:51:47.675580 1155 tablet_bootstrap.cc:492] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.676280 883 tablet_service.cc:1515] Processing DeleteTablet for tablet d7cbc2e6148846f9a58239b089e5a052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.677891 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_47bdd31ed10f4b8183ea24a4fc30d77e" } new_table_name: "default.Table_77b9f4dc97974a79af78d31e6c953f5c" modify_external_catalogs: true
I20250629 01:51:47.680049 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.680191 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba0e267f48a4bf1b58fb2339694719f with delete_type TABLET_DATA_DELETED (replaced by tablet f898483ab9514132be8537ad03720097) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.680400 1156 tablet_replica.cc:331] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:47.680558 1156 raft_consensus.cc:2241] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.680548 1720 raft_consensus.cc:491] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.680712 1156 raft_consensus.cc:2270] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.680680 1720 raft_consensus.cc:513] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.680258 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 with delete_type TABLET_DATA_DELETED (replaced by tablet d73c2eb0d69e467b94a250e1790d8142) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.681257 1720 leader_election.cc:290] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.681730 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da0deb865984f70913323364ca38703" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:47.681908 916 raft_consensus.cc:2466] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:47.681946 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da0deb865984f70913323364ca38703" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.682119 775 raft_consensus.cc:2466] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:47.682287 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d194cd6079254aa794545ca8273a3011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.682468 988 leader_election.cc:304] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913; no voters:
I20250629 01:51:47.682976 1720 raft_consensus.cc:2802] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:47.683049 1720 raft_consensus.cc:491] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:47.683619 1720 raft_consensus.cc:3058] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.685003 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_f07aa6eb42b6478781eb5a65dea53212 [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 6 got 0
I20250629 01:51:47.685897 1720 raft_consensus.cc:513] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.686527 1720 leader_election.cc:290] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.680434 1154 tablet_bootstrap.cc:492] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.687451 745 tablet_service.cc:1468] Processing CreateTablet for tablet 44ab886d05e0468ab0e7b156a32dd392 (DEFAULT_TABLE table=default.Table_d2e4a152a81f434eb4a24f2dd01af367 [id=35de30bc6f17483eb80fef1bf8ce6824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.687727 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ab886d05e0468ab0e7b156a32dd392. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.687983 1154 tablet_bootstrap.cc:654] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.688537 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 24dd74d425a74e79894f309117372ebf (DEFAULT_TABLE table=default.Table_21ecde86214e4cf4a564659ff667e485 [id=9365fab5635a4b18be116e331c4dfda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.688917 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24dd74d425a74e79894f309117372ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.689208 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_37e5f93dba384c5c930e4028ae2a8110"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.689528 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_65f2522f26a440dca7e88a9df18ee4e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.689680 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37e5f93dba384c5c930e4028ae2a8110 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:47.689904 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65f2522f26a440dca7e88a9df18ee4e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.691800 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da0deb865984f70913323364ca38703" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:47.691980 775 raft_consensus.cc:3058] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.692152 739 tablet_service.cc:1468] Processing CreateTablet for tablet 1e4a173d9e70453c91d8ff79a710bc00 (DEFAULT_TABLE table=default.Table_53f9893ad8274a76ad3cea4fd31594c9 [id=6a23a4c8cd70412fa9957be6c583be80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.692466 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e4a173d9e70453c91d8ff79a710bc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.694557 775 raft_consensus.cc:2466] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:47.694912 986 leader_election.cc:304] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:47.695405 1158 ts_tablet_manager.cc:1918] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.695504 1158 log.cc:1199] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:47.695791 1158 ts_tablet_manager.cc:1939] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.696151 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 with delete_type TABLET_DATA_DELETED (replaced by tablet d73c2eb0d69e467b94a250e1790d8142) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.696305 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.696420 740 tablet_service.cc:1515] Processing DeleteTablet for tablet d7cbc2e6148846f9a58239b089e5a052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.697026 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d37334db2b9465a8fcf5e849d5d3b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.699445 1720 raft_consensus.cc:2802] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:47.699584 1720 raft_consensus.cc:695] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:47.699790 1720 consensus_queue.cc:237] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.701283 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7eb02d3ff1104f6885ea1736852bbe57 (DEFAULT_TABLE table=default.Table_7d330f73937f409590ae615703808e30 [id=8f07c83e83e945c8ac69e7e227a35231]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.701614 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb02d3ff1104f6885ea1736852bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.702049 1156 ts_tablet_manager.cc:1905] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.702070 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f (table default.Table_5f4d9b39ee1546eca5a1fb0701b2b93f [id=d5edb2619f57401898104c5d2d036b33]) successfully deleted
W20250629 01:51:47.702287 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a40c1a8eec9549a8bdf695d6516440a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a40c1a8eec9549a8bdf695d6516440a8 already in progress: creating tablet
I20250629 01:51:47.702543 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 0665d3f1225c44f894681fef4ae2520b (DEFAULT_TABLE table=default.Table_35d223e2a4d24c85a2a3780651c205e2 [id=dc9457f3861842c2bbc07463fb367c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.702837 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0665d3f1225c44f894681fef4ae2520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.703801 746 tablet_service.cc:1468] Processing CreateTablet for tablet 7eb02d3ff1104f6885ea1736852bbe57 (DEFAULT_TABLE table=default.Table_7d330f73937f409590ae615703808e30 [id=8f07c83e83e945c8ac69e7e227a35231]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.704123 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb02d3ff1104f6885ea1736852bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.704432 1689 raft_consensus.cc:491] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.704566 1689 raft_consensus.cc:513] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.704841 740 tablet_service.cc:1468] Processing CreateTablet for tablet 24dd74d425a74e79894f309117372ebf (DEFAULT_TABLE table=default.Table_21ecde86214e4cf4a564659ff667e485 [id=9365fab5635a4b18be116e331c4dfda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.705122 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24dd74d425a74e79894f309117372ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.705119 1689 leader_election.cc:290] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.705849 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ba0e267f48a4bf1b58fb2339694719f" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.705960 742 tablet_service.cc:1468] Processing CreateTablet for tablet 0665d3f1225c44f894681fef4ae2520b (DEFAULT_TABLE table=default.Table_35d223e2a4d24c85a2a3780651c205e2 [id=dc9457f3861842c2bbc07463fb367c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.706236 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0665d3f1225c44f894681fef4ae2520b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:47.706262 850 leader_election.cc:343] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: Tablet not RUNNING: STOPPED
W20250629 01:51:47.706439 850 leader_election.cc:343] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 3ba0e267f48a4bf1b58fb2339694719f
I20250629 01:51:47.706460 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu pid 494
I20250629 01:51:47.706543 850 leader_election.cc:304] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.706831 1689 raft_consensus.cc:2747] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.707297 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250629 01:51:47.707347 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da0deb865984f70913323364ca38703" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:47.707504 916 raft_consensus.cc:3058] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.709738 916 raft_consensus.cc:2466] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:47.709895 1155 tablet_bootstrap.cc:654] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.710052 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f7289169dcdb4d8e8dabc33d5e4c53a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.710475 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7289169dcdb4d8e8dabc33d5e4c53a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.712910 741 tablet_service.cc:1468] Processing CreateTablet for tablet 35a62c3fe4fb4220838deece8e46d074 (DEFAULT_TABLE table=default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.713233 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a62c3fe4fb4220838deece8e46d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.713199 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 35a62c3fe4fb4220838deece8e46d074 (DEFAULT_TABLE table=default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.713486 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a62c3fe4fb4220838deece8e46d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.713678 743 tablet_service.cc:1468] Processing CreateTablet for tablet 9988469193514fa9bef285dc18d99ac1 (DEFAULT_TABLE table=default.Table_b86276d79e6a4da3859cdc3c35b24164 [id=7a563c50eb6a450a92e317c7fae3551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.713969 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9988469193514fa9bef285dc18d99ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.714669 1364 raft_consensus.cc:491] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.714823 1364 raft_consensus.cc:513] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.715338 1364 leader_election.cc:290] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:47.716028 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "955a41750b6a4766bb4fd3f01cfd99c8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.716349 738 tablet_service.cc:1468] Processing CreateTablet for tablet c40a07f50971493cb353ed94e850719a (DEFAULT_TABLE table=default.Table_d9cb1a00d22843f69edc93ba305abaad [id=6153bb5c750f4685a0bd20a49447ba87]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:47.716403 986 leader_election.cc:343] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.716619 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40a07f50971493cb353ed94e850719a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.716681 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c40a07f50971493cb353ed94e850719a (DEFAULT_TABLE table=default.Table_d9cb1a00d22843f69edc93ba305abaad [id=6153bb5c750f4685a0bd20a49447ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.716945 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40a07f50971493cb353ed94e850719a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.720610 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_57c0327ef1554e008fc2b4cbc62b28f9" } modify_external_catalogs: true
I20250629 01:51:47.720780 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_57c0327ef1554e008fc2b4cbc62b28f9" } modify_external_catalogs: true
I20250629 01:51:47.721228 747 tablet_service.cc:1468] Processing CreateTablet for tablet bf676660c9b44ffd87e365287e6fc0a9 (DEFAULT_TABLE table=default.Table_78723e0db17d407aaed94a045aac54c1 [id=16a9706229c742f48d50fed46aca558b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.721308 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9988469193514fa9bef285dc18d99ac1 (DEFAULT_TABLE table=default.Table_b86276d79e6a4da3859cdc3c35b24164 [id=7a563c50eb6a450a92e317c7fae3551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.721542 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf676660c9b44ffd87e365287e6fc0a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.721573 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9988469193514fa9bef285dc18d99ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.725216 1016 tablet_service.cc:1468] Processing CreateTablet for tablet e16e6adfa2be40a2885267800331f664 (DEFAULT_TABLE table=default.Table_9587986b83104f2aabf6ef98ebad9e37 [id=fc3e5eca7c8f441dbba114dcd6e82e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.725556 1013 tablet_service.cc:1468] Processing CreateTablet for tablet a53d9e8b7efe46099821ae628640a42e (DEFAULT_TABLE table=default.Table_dfa935fc642246518e47197bd7dda059 [id=85132cd2a72e43b7b3814627e2eb9b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.726497 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16e6adfa2be40a2885267800331f664. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.726742 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53d9e8b7efe46099821ae628640a42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.728235 743 tablet_service.cc:1468] Processing CreateTablet for tablet a53d9e8b7efe46099821ae628640a42e (DEFAULT_TABLE table=default.Table_dfa935fc642246518e47197bd7dda059 [id=85132cd2a72e43b7b3814627e2eb9b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.728665 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53d9e8b7efe46099821ae628640a42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.732481 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 86099eb9e40a4ea389afc059f90a97ef (DEFAULT_TABLE table=default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.732754 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86099eb9e40a4ea389afc059f90a97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.732704 755 tablet_service.cc:1468] Processing CreateTablet for tablet 119248ade66640299a7f17fb158a3a5a (DEFAULT_TABLE table=default.Table_7e58c9f288004df185e961b29ae19383 [id=78751011b87b4d92acbc14bb7025559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.733000 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119248ade66640299a7f17fb158a3a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.733541 1014 tablet_service.cc:1468] Processing CreateTablet for tablet bf676660c9b44ffd87e365287e6fc0a9 (DEFAULT_TABLE table=default.Table_78723e0db17d407aaed94a045aac54c1 [id=16a9706229c742f48d50fed46aca558b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.734427 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5da0f7f4f413447e84b57be34c67019b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.734822 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5da0f7f4f413447e84b57be34c67019b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.735682 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_415b32694788453db597034d0865a18a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.735847 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dcfccdde4f3941dd8db4b2db608ed1d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.736122 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_415b32694788453db597034d0865a18a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:47.736294 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcfccdde4f3941dd8db4b2db608ed1d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.737712 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf676660c9b44ffd87e365287e6fc0a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.738369 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_eacbe3d626eb4bb980d04c2c6647aec9" } modify_external_catalogs: true
I20250629 01:51:47.738515 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_eacbe3d626eb4bb980d04c2c6647aec9" } modify_external_catalogs: true
I20250629 01:51:47.739229 747 tablet_service.cc:1468] Processing CreateTablet for tablet 86099eb9e40a4ea389afc059f90a97ef (DEFAULT_TABLE table=default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.739502 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86099eb9e40a4ea389afc059f90a97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.740830 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_edae10908319498e8780ce62b5f260c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.741199 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edae10908319498e8780ce62b5f260c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.742134 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b070f6a145e04ae4b3bcf620e35d2943" } new_table_name: "default.Table_296cbd48cdc743689498e69b6d8c1a7c" modify_external_catalogs: true
I20250629 01:51:47.743206 448 master_service.cc:948] ReplaceTablet: received request to replace tablet bccdcd9de08648bb925db6fe56f97d20 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:47.747094 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f6fd2631c6241c6b6e205ccf4a313df" } new_table_name: "default.Table_b15fad4b4b3c403daa112af034e6e3d8" modify_external_catalogs: true
W20250629 01:51:47.747247 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1ea681bd91e54e4898bcfb72bed6198f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea681bd91e54e4898bcfb72bed6198f already in progress: creating tablet
I20250629 01:51:47.747253 752 tablet_service.cc:1468] Processing CreateTablet for tablet 556c2050d2cb40f28a3a0bfb9d677d69 (DEFAULT_TABLE table=default.Table_93358968e4994116ad323cf7bdd275d3 [id=060f3d672f764df5897804364e4b1bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.747524 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556c2050d2cb40f28a3a0bfb9d677d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.748255 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 119248ade66640299a7f17fb158a3a5a (DEFAULT_TABLE table=default.Table_7e58c9f288004df185e961b29ae19383 [id=78751011b87b4d92acbc14bb7025559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.748560 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119248ade66640299a7f17fb158a3a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.751883 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 556c2050d2cb40f28a3a0bfb9d677d69 (DEFAULT_TABLE table=default.Table_93358968e4994116ad323cf7bdd275d3 [id=060f3d672f764df5897804364e4b1bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.752215 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5f60c48c924d49a186a8c81d154bdc54 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:40 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.716496 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "955a41750b6a4766bb4fd3f01cfd99c8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:47.752854 988 leader_election.cc:343] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.753000 988 leader_election.cc:304] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.753542 1364 raft_consensus.cc:2747] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.753881 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556c2050d2cb40f28a3a0bfb9d677d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.757028 1690 tablet.cc:1722] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:47.759091 1681 tablet.cc:1722] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:47.760215 1682 tablet.cc:1722] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:47.762318 1156 ts_tablet_manager.cc:1918] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.762416 1156 log.cc:1199] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/3ba0e267f48a4bf1b58fb2339694719f
I20250629 01:51:47.762766 1156 ts_tablet_manager.cc:1939] T 3ba0e267f48a4bf1b58fb2339694719f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:47.763669 744 tablet_service.cc:1468] Processing CreateTablet for tablet e16e6adfa2be40a2885267800331f664 (DEFAULT_TABLE table=default.Table_9587986b83104f2aabf6ef98ebad9e37 [id=fc3e5eca7c8f441dbba114dcd6e82e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.765203 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16e6adfa2be40a2885267800331f664. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.766710 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.777410 1156 tablet_replica.cc:331] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:47.777624 1156 raft_consensus.cc:2241] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.777768 1156 raft_consensus.cc:2270] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.782603 1156 ts_tablet_manager.cc:1905] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.786607 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2d37334db2b9465a8fcf5e849d5d3b0c (DEFAULT_TABLE table=default.Table_3827f711f3174f66b469302de8091583 [id=e11fdfa462d54c6c83b5582fd119a925]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.787034 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d37334db2b9465a8fcf5e849d5d3b0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.789500 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 3ba0e267f48a4bf1b58fb2339694719f (table default.Table_3abcbdb38a38484782a28269e7b69beb [id=29b0e64154664e928993c1035a1bb23c]) successfully deleted
W20250629 01:51:47.789790 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 56b60431b72e4792aa1c32fdee40cdb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56b60431b72e4792aa1c32fdee40cdb9 already in progress: creating tablet
W20250629 01:51:47.790015 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a40c1a8eec9549a8bdf695d6516440a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a40c1a8eec9549a8bdf695d6516440a8 already in progress: creating tablet
W20250629 01:51:47.790220 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d7cbc2e6148846f9a58239b089e5a052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7cbc2e6148846f9a58239b089e5a052 already in progress: creating tablet
I20250629 01:51:47.794905 1158 tablet_replica.cc:331] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:47.795428 1158 raft_consensus.cc:2241] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:47.795969 1158 raft_consensus.cc:2270] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:47.797716 1155 tablet_bootstrap.cc:492] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.798403 1155 ts_tablet_manager.cc:1397] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.122s user 0.000s sys 0.005s
I20250629 01:51:47.799091 1155 raft_consensus.cc:357] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.799283 1155 raft_consensus.cc:383] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.799352 1155 raft_consensus.cc:738] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.799599 1155 consensus_queue.cc:260] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.800241 1155 ts_tablet_manager.cc:1428] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250629 01:51:47.800577 1155 tablet_bootstrap.cc:492] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.804324 1158 ts_tablet_manager.cc:1905] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:47.803428 1155 tablet_bootstrap.cc:654] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.807906 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 959837d39a9a4a6f8735a906b3a53222
I20250629 01:51:47.808956 1156 ts_tablet_manager.cc:1918] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.809325 1156 log.cc:1199] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5f60c48c924d49a186a8c81d154bdc54
I20250629 01:51:47.809617 1156 ts_tablet_manager.cc:1939] T 5f60c48c924d49a186a8c81d154bdc54 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:47.809859 1154 tablet_bootstrap.cc:492] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.810020 1154 ts_tablet_manager.cc:1397] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.130s user 0.005s sys 0.005s
I20250629 01:51:47.810757 1154 raft_consensus.cc:357] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.810971 1154 raft_consensus.cc:383] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.811060 1154 raft_consensus.cc:738] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.811270 1154 consensus_queue.cc:260] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.811721 1154 ts_tablet_manager.cc:1428] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:47.811992 1154 tablet_bootstrap.cc:492] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.814282 1154 tablet_bootstrap.cc:654] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.816308 1155 tablet_bootstrap.cc:492] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.816417 1155 ts_tablet_manager.cc:1397] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20250629 01:51:47.817008 1155 raft_consensus.cc:357] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.817162 1155 raft_consensus.cc:383] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.817276 1155 raft_consensus.cc:738] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.817467 1155 consensus_queue.cc:260] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.817945 1155 ts_tablet_manager.cc:1428] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:47.818243 1155 tablet_bootstrap.cc:492] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.821897 1155 tablet_bootstrap.cc:654] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.823745 883 tablet_service.cc:1468] Processing CreateTablet for tablet 7eb02d3ff1104f6885ea1736852bbe57 (DEFAULT_TABLE table=default.Table_7d330f73937f409590ae615703808e30 [id=8f07c83e83e945c8ac69e7e227a35231]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.825660 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb02d3ff1104f6885ea1736852bbe57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.827035 892 tablet_service.cc:1468] Processing CreateTablet for tablet c40a07f50971493cb353ed94e850719a (DEFAULT_TABLE table=default.Table_d9cb1a00d22843f69edc93ba305abaad [id=6153bb5c750f4685a0bd20a49447ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.827314 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40a07f50971493cb353ed94e850719a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.828104 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9988469193514fa9bef285dc18d99ac1 (DEFAULT_TABLE table=default.Table_b86276d79e6a4da3859cdc3c35b24164 [id=7a563c50eb6a450a92e317c7fae3551e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.828368 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9988469193514fa9bef285dc18d99ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.827931 886 tablet_service.cc:1468] Processing CreateTablet for tablet 24dd74d425a74e79894f309117372ebf (DEFAULT_TABLE table=default.Table_21ecde86214e4cf4a564659ff667e485 [id=9365fab5635a4b18be116e331c4dfda7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.829282 893 tablet_service.cc:1468] Processing CreateTablet for tablet a53d9e8b7efe46099821ae628640a42e (DEFAULT_TABLE table=default.Table_dfa935fc642246518e47197bd7dda059 [id=85132cd2a72e43b7b3814627e2eb9b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.830224 885 tablet_service.cc:1468] Processing CreateTablet for tablet bf676660c9b44ffd87e365287e6fc0a9 (DEFAULT_TABLE table=default.Table_78723e0db17d407aaed94a045aac54c1 [id=16a9706229c742f48d50fed46aca558b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.830348 881 tablet_service.cc:1468] Processing CreateTablet for tablet 86099eb9e40a4ea389afc059f90a97ef (DEFAULT_TABLE table=default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.832404 880 tablet_service.cc:1468] Processing CreateTablet for tablet 0665d3f1225c44f894681fef4ae2520b (DEFAULT_TABLE table=default.Table_35d223e2a4d24c85a2a3780651c205e2 [id=dc9457f3861842c2bbc07463fb367c0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.832684 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 959837d39a9a4a6f8735a906b3a53222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.833243 887 tablet_service.cc:1468] Processing CreateTablet for tablet e16e6adfa2be40a2885267800331f664 (DEFAULT_TABLE table=default.Table_9587986b83104f2aabf6ef98ebad9e37 [id=fc3e5eca7c8f441dbba114dcd6e82e78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.833475 1158 ts_tablet_manager.cc:1918] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:47.833488 894 tablet_service.cc:1468] Processing CreateTablet for tablet 556c2050d2cb40f28a3a0bfb9d677d69 (DEFAULT_TABLE table=default.Table_93358968e4994116ad323cf7bdd275d3 [id=060f3d672f764df5897804364e4b1bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.833592 1158 log.cc:1199] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d61687e19d994e87b09e55bd3b9476d7
I20250629 01:51:47.833722 888 tablet_service.cc:1468] Processing CreateTablet for tablet 119248ade66640299a7f17fb158a3a5a (DEFAULT_TABLE table=default.Table_7e58c9f288004df185e961b29ae19383 [id=78751011b87b4d92acbc14bb7025559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.833882 1158 ts_tablet_manager.cc:1939] T d61687e19d994e87b09e55bd3b9476d7 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:47.833962 889 tablet_service.cc:1468] Processing CreateTablet for tablet 35a62c3fe4fb4220838deece8e46d074 (DEFAULT_TABLE table=default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:47.834331 1155 tablet_bootstrap.cc:492] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.834441 1155 ts_tablet_manager.cc:1397] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20250629 01:51:47.834887 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a53d9e8b7efe46099821ae628640a42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.835537 1155 raft_consensus.cc:357] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.835678 1155 raft_consensus.cc:383] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.835747 1155 raft_consensus.cc:738] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.835935 1155 consensus_queue.cc:260] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.836458 1155 ts_tablet_manager.cc:1428] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:47.836863 1155 tablet_bootstrap.cc:492] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.837477 1154 tablet_bootstrap.cc:492] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.837591 1154 ts_tablet_manager.cc:1397] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20250629 01:51:47.838207 1154 raft_consensus.cc:357] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.838347 1154 raft_consensus.cc:383] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.838416 1154 raft_consensus.cc:738] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.838578 1154 consensus_queue.cc:260] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.838989 1154 ts_tablet_manager.cc:1428] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:47.839223 1154 tablet_bootstrap.cc:492] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.839318 1155 tablet_bootstrap.cc:654] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.840619 1154 tablet_bootstrap.cc:654] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.841424 1155 tablet_bootstrap.cc:492] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:47.841593 1155 ts_tablet_manager.cc:1397] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:47.842720 1155 raft_consensus.cc:357] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.842936 1155 raft_consensus.cc:383] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.843040 1155 raft_consensus.cc:738] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.843993 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a62c3fe4fb4220838deece8e46d074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.843276 1155 consensus_queue.cc:260] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.844664 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86099eb9e40a4ea389afc059f90a97ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.844965 1155 ts_tablet_manager.cc:1428] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:47.845177 1155 tablet_bootstrap.cc:492] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:47.845377 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 119248ade66640299a7f17fb158a3a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.845886 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556c2050d2cb40f28a3a0bfb9d677d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.846347 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0665d3f1225c44f894681fef4ae2520b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.846350 1155 tablet_bootstrap.cc:654] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.847023 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf676660c9b44ffd87e365287e6fc0a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.847548 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e16e6adfa2be40a2885267800331f664. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.866643 1154 tablet_bootstrap.cc:492] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.866770 1154 ts_tablet_manager.cc:1397] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20250629 01:51:47.867358 1154 raft_consensus.cc:357] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.867506 1154 raft_consensus.cc:383] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.867552 1154 raft_consensus.cc:738] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.867748 1154 consensus_queue.cc:260] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.868180 1154 ts_tablet_manager.cc:1428] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:47.868505 1154 tablet_bootstrap.cc:492] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.871013 1154 tablet_bootstrap.cc:654] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:47.883364 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d6b6d5ff92c343cb8f1b3039b84390af (table default.Table_fb0c134a9a1548cd8cebc3508bde937a [id=8038bbd131674ed59dbe36c73c14d466], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet d6b6d5ff92c343cb8f1b3039b84390af consensus state has no leader
I20250629 01:51:47.892943 1154 tablet_bootstrap.cc:492] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.893102 1154 ts_tablet_manager.cc:1397] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20250629 01:51:47.893923 1154 raft_consensus.cc:357] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.894121 1154 raft_consensus.cc:383] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.894210 1154 raft_consensus.cc:738] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.894423 1154 consensus_queue.cc:260] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.895002 1154 ts_tablet_manager.cc:1428] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:47.895282 1154 tablet_bootstrap.cc:492] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.897209 1154 tablet_bootstrap.cc:654] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.900755 1364 raft_consensus.cc:491] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.900876 1364 raft_consensus.cc:513] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.901394 1364 leader_election.cc:290] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:47.902134 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "881461aa1bc34dd483f65e171c341bc5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:47.902822 986 leader_election.cc:343] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.905251 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "881461aa1bc34dd483f65e171c341bc5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:47.905658 988 leader_election.cc:343] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.905758 988 leader_election.cc:304] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.906298 1720 raft_consensus.cc:2747] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.906535 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24dd74d425a74e79894f309117372ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.911267 1154 tablet_bootstrap.cc:492] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.911397 1154 ts_tablet_manager.cc:1397] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.006s
I20250629 01:51:47.911926 1154 raft_consensus.cc:357] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.912082 1154 raft_consensus.cc:383] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.912158 1154 raft_consensus.cc:738] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.912355 1154 consensus_queue.cc:260] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.912989 1154 ts_tablet_manager.cc:1428] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:47.913288 1154 tablet_bootstrap.cc:492] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.914469 1154 tablet_bootstrap.cc:654] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.922288 1154 tablet_bootstrap.cc:492] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.922434 1154 ts_tablet_manager.cc:1397] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20250629 01:51:47.923280 1154 raft_consensus.cc:357] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.923466 1154 raft_consensus.cc:383] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.923563 1154 raft_consensus.cc:738] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.923820 1154 consensus_queue.cc:260] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.924407 1154 ts_tablet_manager.cc:1428] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:47.924907 1154 tablet_bootstrap.cc:492] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.926750 1154 tablet_bootstrap.cc:654] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:47.927768 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 56b60431b72e4792aa1c32fdee40cdb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56b60431b72e4792aa1c32fdee40cdb9 already in progress: creating tablet
W20250629 01:51:47.938699 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d7cbc2e6148846f9a58239b089e5a052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7cbc2e6148846f9a58239b089e5a052 already in progress: creating tablet
W20250629 01:51:47.939189 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1ea681bd91e54e4898bcfb72bed6198f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea681bd91e54e4898bcfb72bed6198f already in progress: creating tablet
I20250629 01:51:47.939597 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d61687e19d994e87b09e55bd3b9476d7 (table default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb [id=f274d40e73944253825a87b39ee56b71]) successfully deleted
I20250629 01:51:47.940428 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 959837d39a9a4a6f8735a906b3a53222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.940701 1154 tablet_bootstrap.cc:492] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.940876 1154 ts_tablet_manager.cc:1397] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.002s
I20250629 01:51:47.941488 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 959837d39a9a4a6f8735a906b3a53222 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.942030 1154 raft_consensus.cc:357] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.942294 1154 raft_consensus.cc:383] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.942411 1154 raft_consensus.cc:738] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.942718 1154 consensus_queue.cc:260] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.943305 1154 ts_tablet_manager.cc:1428] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:47.944401 1159 tablet_bootstrap.cc:492] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:47.944547 1159 ts_tablet_manager.cc:1397] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.308s user 0.001s sys 0.003s
I20250629 01:51:47.944857 1154 tablet_bootstrap.cc:492] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.946841 1154 tablet_bootstrap.cc:654] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.947638 1159 raft_consensus.cc:357] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.947795 1159 raft_consensus.cc:383] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.948685 775 raft_consensus.cc:3058] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:47.948978 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5f60c48c924d49a186a8c81d154bdc54 (table default.Table_376190835c644f499104ffbbd085ac10 [id=85d2dfba5d5c422a9631994a62cfe443]) successfully deleted
W20250629 01:51:47.949177 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 959837d39a9a4a6f8735a906b3a53222 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 959837d39a9a4a6f8735a906b3a53222 already in progress: creating tablet
I20250629 01:51:47.950321 915 raft_consensus.cc:1273] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:47.951333 775 raft_consensus.cc:1273] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:47.951807 1720 consensus_queue.cc:1035] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:47.952729 1720 consensus_queue.cc:1035] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:47.953491 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 67e90a48af8f48bba3b3fd4f4110d367
I20250629 01:51:47.954031 446 master_service.cc:948] ReplaceTablet: received request to replace tablet cb7314fbdfc74b1c8d9d1be9c05244cb from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:47.956308 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b0ef1ba46ac487ab1502c3d116814e1 consensus state has no leader
I20250629 01:51:47.957645 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bccdcd9de08648bb925db6fe56f97d20
I20250629 01:51:47.957978 448 catalog_manager.cc:6652] ReplaceTablet: tablet bccdcd9de08648bb925db6fe56f97d20 deleted and replaced by tablet fb48f4a2d9724300b9c00df6e12e92b0
W20250629 01:51:47.959880 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbe1f8083a884cdd97afce83bd22073c consensus state has no leader
I20250629 01:51:47.960943 1154 tablet_bootstrap.cc:492] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.961078 1154 ts_tablet_manager.cc:1397] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250629 01:51:47.960981 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0cf9b58194d14b1a8f94c2c12e54a36b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:47.961445 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cf9b58194d14b1a8f94c2c12e54a36b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.961656 1154 raft_consensus.cc:357] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.961778 1154 raft_consensus.cc:383] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.961848 1154 raft_consensus.cc:738] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.962046 1154 consensus_queue.cc:260] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.962424 1154 ts_tablet_manager.cc:1428] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:47.962709 1154 tablet_bootstrap.cc:492] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.963121 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8ace6d33743f4b9aa9a7ec9e11f5a0b6" } new_table_name: "default.Table_589cda708d5c486cb3cf4cee7b376856" modify_external_catalogs: true
I20250629 01:51:47.963747 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b8599c95728d43dd898fbdfcb4fe5db3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:47.964191 1154 tablet_bootstrap.cc:654] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:47.964177 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8599c95728d43dd898fbdfcb4fe5db3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:47.964483 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7e58c9f288004df185e961b29ae19383" } modify_external_catalogs: true
I20250629 01:51:47.964648 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7e58c9f288004df185e961b29ae19383" } modify_external_catalogs: true
I20250629 01:51:47.965143 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1b37a535cd2f483a8c72aa10882a4301" } modify_external_catalogs: true
I20250629 01:51:47.965301 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1b37a535cd2f483a8c72aa10882a4301" } modify_external_catalogs: true
I20250629 01:51:47.966987 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55cf597f7eab480299bb4d32ccf5868c" } modify_external_catalogs: true
I20250629 01:51:47.967171 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55cf597f7eab480299bb4d32ccf5868c" } modify_external_catalogs: true
I20250629 01:51:47.967878 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ea3a09c86c72411e97c6d80a7c665599" } new_table_name: "default.Table_1f1cf7aea79d46d9998b288e0c782dd6" modify_external_catalogs: true
I20250629 01:51:47.969883 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.976861 1154 tablet_bootstrap.cc:492] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.977041 1154 ts_tablet_manager.cc:1397] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250629 01:51:47.977783 1154 raft_consensus.cc:357] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.978001 1154 raft_consensus.cc:383] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.978082 1154 raft_consensus.cc:738] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.978291 880 tablet_service.cc:1515] Processing DeleteTablet for tablet bccdcd9de08648bb925db6fe56f97d20 with delete_type TABLET_DATA_DELETED (replaced by tablet fb48f4a2d9724300b9c00df6e12e92b0) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:47.978461 1159 raft_consensus.cc:738] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.978375 1154 consensus_queue.cc:260] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.978905 1154 ts_tablet_manager.cc:1428] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:47.980515 1154 tablet_bootstrap.cc:492] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:47.980911 1159 consensus_queue.cc:260] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:47.982194 1154 tablet_bootstrap.cc:654] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:47.982343 1157 ts_tablet_manager.cc:1918] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250629 01:51:47.986361 1720 raft_consensus.cc:491] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.986536 1157 log.cc:1199] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a72c6040421642578fac307ea9914ef6
I20250629 01:51:47.986495 1720 raft_consensus.cc:513] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.987087 1720 leader_election.cc:290] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.987751 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69980d5bf37d4001a6196b22c34ccc94" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:47.987893 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69980d5bf37d4001a6196b22c34ccc94" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:47.988343 986 leader_election.cc:343] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:47.988606 988 leader_election.cc:343] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:47.988708 988 leader_election.cc:304] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.989238 1720 raft_consensus.cc:2747] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:47.989729 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 959837d39a9a4a6f8735a906b3a53222 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 959837d39a9a4a6f8735a906b3a53222 already in progress: creating tablet
I20250629 01:51:47.989923 1159 ts_tablet_manager.cc:1428] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.044s user 0.000s sys 0.002s
I20250629 01:51:47.990585 1159 tablet_bootstrap.cc:492] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:47.991263 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:47.990262 1157 ts_tablet_manager.cc:1939] T a72c6040421642578fac307ea9914ef6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:47.991971 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 538) took 1344 ms. Trace:
I20250629 01:51:47.992491 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.995098 743 tablet_service.cc:1515] Processing DeleteTablet for tablet bccdcd9de08648bb925db6fe56f97d20 with delete_type TABLET_DATA_DELETED (replaced by tablet fb48f4a2d9724300b9c00df6e12e92b0) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:47.995620 1689 raft_consensus.cc:491] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:47.997684 1689 raft_consensus.cc:513] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:47.998092 1154 tablet_bootstrap.cc:492] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:47.998196 1689 leader_election.cc:290] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:47.998224 1154 ts_tablet_manager.cc:1397] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20250629 01:51:47.998863 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:47.998901 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2562570cb0846e19ebc97bdb43bd70d" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:47.998961 1154 raft_consensus.cc:357] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:47.999099 1154 raft_consensus.cc:383] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:47.999183 1154 raft_consensus.cc:738] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:47.999404 850 leader_election.cc:304] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:47.999614 1689 raft_consensus.cc:2747] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:47.999544 1154 consensus_queue.cc:260] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.000129 1154 ts_tablet_manager.cc:1428] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.000298 1157 rpcz_store.cc:276] 0629 01:51:46.647201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.681809 (+ 34608us) service_pool.cc:225] Handling call
0629 01:51:47.568634 (+886825us) ts_tablet_manager.cc:1145] Deleting tablet a72c6040421642578fac307ea9914ef6
0629 01:51:47.568642 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:47.779112 (+210470us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.949697 (+170585us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:47.991922 (+ 42225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31074,"tablet-delete.queue_time_us":886656}
I20250629 01:51:48.000463 1154 tablet_bootstrap.cc:492] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.001201 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet bccdcd9de08648bb925db6fe56f97d20 with delete_type TABLET_DATA_DELETED (replaced by tablet fb48f4a2d9724300b9c00df6e12e92b0) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.002480 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 36fcb819c61c4fabbb0da295d0002998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.002727 1156 tablet_replica.cc:331] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.002899 1156 raft_consensus.cc:2241] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.003098 1156 raft_consensus.cc:2270] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.003628 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a72c6040421642578fac307ea9914ef6 (table default.Table_eab06a5f9cbc4248a1757e3b5518545b [id=6a41ce2e3dcf43ce9edb6392a244d9a2]) successfully deleted
I20250629 01:51:48.004038 1156 ts_tablet_manager.cc:1905] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.004400 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 539) took 1357 ms. Trace:
I20250629 01:51:48.012609 1156 ts_tablet_manager.cc:1918] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.012746 1156 log.cc:1199] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/36fcb819c61c4fabbb0da295d0002998
I20250629 01:51:48.013181 1156 ts_tablet_manager.cc:1939] T 36fcb819c61c4fabbb0da295d0002998 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.003160 1154 tablet_bootstrap.cc:654] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.008378 1720 raft_consensus.cc:491] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.015267 1720 raft_consensus.cc:513] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.015756 1720 leader_election.cc:290] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:48.012024 1681 tablet.cc:1722] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:48.020874 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dfe921a84694903bbdb3fed3d6f2761" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.021080 773 raft_consensus.cc:2466] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:48.021495 986 leader_election.cc:304] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:48.021788 1720 raft_consensus.cc:2802] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:48.021909 1720 raft_consensus.cc:491] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:48.022059 1720 raft_consensus.cc:3058] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.023150 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_77b9f4dc97974a79af78d31e6c953f5c [id=e6d6685d64154ba8ab80acf601e83986] alter complete (version 1)
I20250629 01:51:48.024596 1720 raft_consensus.cc:513] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:48.024796 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fcead51fd9b43d0a52c4669a5e23fe5 consensus state has no leader
I20250629 01:51:48.025456 1154 tablet_bootstrap.cc:492] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.025457 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dfe921a84694903bbdb3fed3d6f2761" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:48.025630 1154 ts_tablet_manager.cc:1397] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20250629 01:51:48.025627 773 raft_consensus.cc:3058] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.026425 1154 raft_consensus.cc:357] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.026633 1154 raft_consensus.cc:383] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.026705 1154 raft_consensus.cc:738] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:48.026898 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3070ce8c467a47b0a833374450911291 consensus state has no leader
I20250629 01:51:48.027016 1154 consensus_queue.cc:260] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.027146 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 5dfe921a84694903bbdb3fed3d6f2761 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.028024 1154 ts_tablet_manager.cc:1428] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.028373 1154 tablet_bootstrap.cc:492] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.028326 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f7289169dcdb4d8e8dabc33d5e4c53a9" } new_table_name: "default.Table_14ca2bdaa17c48b59043ada398aa4bc4" modify_external_catalogs: true
I20250629 01:51:48.028913 773 raft_consensus.cc:2466] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:48.029326 986 leader_election.cc:304] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:48.029621 1710 raft_consensus.cc:2802] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:48.029531 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_951aa64d945448d4b3c17187cf5bff48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.029745 1710 raft_consensus.cc:695] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:48.029984 1710 consensus_queue.cc:237] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.030663 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_705f111850114af4b8ada44b31486222"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.031116 1154 tablet_bootstrap.cc:654] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.031512 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c5c42b0786ae4ba8ae6d0165aae1bc60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.031889 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5c42b0786ae4ba8ae6d0165aae1bc60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:48.033344 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_951aa64d945448d4b3c17187cf5bff48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.034323 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1c68a0f197b241f781976418aed57d90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.034605 1720 leader_election.cc:290] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:48.034770 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c68a0f197b241f781976418aed57d90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:48.036175 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_705f111850114af4b8ada44b31486222 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.014322 1690 tablet.cc:1722] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:48.040169 1157 rpcz_store.cc:276] 0629 01:51:46.647288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.681992 (+ 34704us) service_pool.cc:225] Handling call
0629 01:51:48.004265 (+1322273us) ts_tablet_manager.cc:1145] Deleting tablet a75d7986152f427d88ed34a443128bc6
0629 01:51:48.004272 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.004361 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321704}
I20250629 01:51:48.020982 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dfe921a84694903bbdb3fed3d6f2761" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.039278 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5dfe921a84694903bbdb3fed3d6f2761" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:48.040952 988 leader_election.cc:343] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:48.041181 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a75d7986152f427d88ed34a443128bc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a75d7986152f427d88ed34a443128bc6 already in progress: creating tablet
W20250629 01:51:48.041292 988 leader_election.cc:343] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.044180 1154 tablet_bootstrap.cc:492] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.044353 1154 ts_tablet_manager.cc:1397] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
W20250629 01:51:48.044644 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 67e90a48af8f48bba3b3fd4f4110d367 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67e90a48af8f48bba3b3fd4f4110d367 already in progress: creating tablet
W20250629 01:51:48.044862 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet bccdcd9de08648bb925db6fe56f97d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bccdcd9de08648bb925db6fe56f97d20 already in progress: creating tablet
I20250629 01:51:48.045054 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 36fcb819c61c4fabbb0da295d0002998 (table default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]) successfully deleted
W20250629 01:51:48.045275 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 67e90a48af8f48bba3b3fd4f4110d367 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67e90a48af8f48bba3b3fd4f4110d367 already in progress: creating tablet
W20250629 01:51:48.045452 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet bccdcd9de08648bb925db6fe56f97d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bccdcd9de08648bb925db6fe56f97d20 already in progress: creating tablet
I20250629 01:51:48.045941 1154 raft_consensus.cc:357] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.046147 1154 raft_consensus.cc:383] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.046249 1154 raft_consensus.cc:738] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.046492 1154 consensus_queue.cc:260] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.047125 1154 ts_tablet_manager.cc:1428] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:48.047463 1154 tablet_bootstrap.cc:492] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.048233 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 537) took 1401 ms. Trace:
I20250629 01:51:48.048319 1157 rpcz_store.cc:276] 0629 01:51:46.647106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.654687 (+ 7581us) service_pool.cc:225] Handling call
0629 01:51:48.040547 (+1385860us) ts_tablet_manager.cc:1145] Deleting tablet eb26725bd175429d805947c4eda6e40a
0629 01:51:48.048068 (+ 7521us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.048179 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1330931}
I20250629 01:51:48.048758 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 541) took 1300 ms. Trace:
I20250629 01:51:48.048830 1157 rpcz_store.cc:276] 0629 01:51:46.747982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.748247 (+ 265us) service_pool.cc:225] Handling call
0629 01:51:48.048635 (+1300388us) ts_tablet_manager.cc:1145] Deleting tablet 327c303b6b9347f5b4cb78f74d41a44a
0629 01:51:48.048644 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.048716 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1298935}
I20250629 01:51:48.049233 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 540) took 1303 ms. Trace:
I20250629 01:51:48.049316 1154 tablet_bootstrap.cc:654] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.049310 1157 rpcz_store.cc:276] 0629 01:51:46.745400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.745468 (+ 68us) service_pool.cc:225] Handling call
0629 01:51:48.049112 (+1303644us) ts_tablet_manager.cc:1145] Deleting tablet 3a81e6f9711c4d398ae09f33a75566ae
0629 01:51:48.049119 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.049182 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1239538}
I20250629 01:51:48.049685 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 543) took 1231 ms. Trace:
I20250629 01:51:48.049762 1157 rpcz_store.cc:276] 0629 01:51:46.818157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.818247 (+ 90us) service_pool.cc:225] Handling call
0629 01:51:48.049569 (+1231322us) ts_tablet_manager.cc:1145] Deleting tablet 6092e19acf974f2dad5a903da72944ba
0629 01:51:48.049575 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.049646 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224799}
W20250629 01:51:48.050065 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet eb26725bd175429d805947c4eda6e40a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb26725bd175429d805947c4eda6e40a already in progress: creating tablet
W20250629 01:51:48.050284 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 327c303b6b9347f5b4cb78f74d41a44a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 327c303b6b9347f5b4cb78f74d41a44a already in progress: creating tablet
W20250629 01:51:48.050467 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3a81e6f9711c4d398ae09f33a75566ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a81e6f9711c4d398ae09f33a75566ae already in progress: creating tablet
W20250629 01:51:48.050674 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6092e19acf974f2dad5a903da72944ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6092e19acf974f2dad5a903da72944ba already in progress: creating tablet
I20250629 01:51:48.050927 1157 tablet_replica.cc:331] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:48.051112 1157 raft_consensus.cc:2241] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.051290 1157 raft_consensus.cc:2270] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.051739 1157 ts_tablet_manager.cc:1905] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.056667 1710 raft_consensus.cc:491] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.056828 1710 raft_consensus.cc:513] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.057880 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ed143377709465d9ae5df7c2ec3ca47" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:48.058565 986 leader_election.cc:343] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.058744 1710 leader_election.cc:290] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.060540 1154 tablet_bootstrap.cc:492] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.060674 1154 ts_tablet_manager.cc:1397] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250629 01:51:48.060925 449 catalog_manager.cc:5582] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:48.061316 1154 raft_consensus.cc:357] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.061481 1154 raft_consensus.cc:383] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.061543 1154 raft_consensus.cc:738] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.061764 1154 consensus_queue.cc:260] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.062327 1154 ts_tablet_manager.cc:1428] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:48.062561 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 (DEFAULT_TABLE table=default.Table_6429e6545a1d4d5bb1b37b6799c105ad [id=e87d9a774eb1499f8c9d91bdf13aa08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.062662 1154 tablet_bootstrap.cc:492] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.062927 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.065304 1154 tablet_bootstrap.cc:654] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.066804 743 tablet_service.cc:1468] Processing CreateTablet for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 (DEFAULT_TABLE table=default.Table_6429e6545a1d4d5bb1b37b6799c105ad [id=e87d9a774eb1499f8c9d91bdf13aa08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.067261 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.067307 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62369ffc1d4a4628bb24254877d6dee6 has no consensus state
I20250629 01:51:48.076512 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 12873ec9a5574e789f9d669ddda85704 (DEFAULT_TABLE table=default.Table_6864f50c0e624776b4090fb5368ae3dc [id=8fb3448eefdc46cbbe3f9d8cc52c6c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.076826 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12873ec9a5574e789f9d669ddda85704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.079012 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_caef1729000a42b482b4da2dcd58f39a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.079396 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_caef1729000a42b482b4da2dcd58f39a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.080487 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8aceaa540cf94e83ba9250c612cab7a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.080922 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8aceaa540cf94e83ba9250c612cab7a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.080925 744 tablet_service.cc:1468] Processing CreateTablet for tablet 12873ec9a5574e789f9d669ddda85704 (DEFAULT_TABLE table=default.Table_6864f50c0e624776b4090fb5368ae3dc [id=8fb3448eefdc46cbbe3f9d8cc52c6c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.081331 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12873ec9a5574e789f9d669ddda85704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.082265 755 tablet_service.cc:1468] Processing CreateTablet for tablet c969377aaf034e9b896c1b53366555ff (DEFAULT_TABLE table=default.Table_cca695fad03947debe4d1d515b3c73d8 [id=9d441f2151974914945282a0e0c4c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.082650 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c969377aaf034e9b896c1b53366555ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.083667 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d7789b77f4814c388e1eb092297120b1" } modify_external_catalogs: true
I20250629 01:51:48.083797 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d7789b77f4814c388e1eb092297120b1" } modify_external_catalogs: true
I20250629 01:51:48.087702 749 tablet_service.cc:1468] Processing CreateTablet for tablet 185cc5f8c17146fa92dda7d31cde4416 (DEFAULT_TABLE table=default.Table_7fed58fcd48d4bd6ba7e1ce4d4c40635 [id=089048e00fbc4e7b8b14e185f89df7f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.088145 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185cc5f8c17146fa92dda7d31cde4416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.089682 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 25f4de297274444982095e9c308f0983 (DEFAULT_TABLE table=default.Table_711c32a531fd46489436ed1a683d3297 [id=c78a485212fc4b19a5b17df637acb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.090955 747 tablet_service.cc:1468] Processing CreateTablet for tablet 25f4de297274444982095e9c308f0983 (DEFAULT_TABLE table=default.Table_711c32a531fd46489436ed1a683d3297 [id=c78a485212fc4b19a5b17df637acb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.091305 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f4de297274444982095e9c308f0983. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.092396 745 tablet_service.cc:1468] Processing CreateTablet for tablet 07d5eb1b3c3345c099f298df240a5c75 (DEFAULT_TABLE table=default.Table_da7726c4aa4a4bf893ec5b19264cd6ea [id=bc368c1f84884f5d988a9b71a78dd65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.092779 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d5eb1b3c3345c099f298df240a5c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.092813 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 07d5eb1b3c3345c099f298df240a5c75 (DEFAULT_TABLE table=default.Table_da7726c4aa4a4bf893ec5b19264cd6ea [id=bc368c1f84884f5d988a9b71a78dd65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.093012 1032 tablet_service.cc:1468] Processing CreateTablet for tablet c969377aaf034e9b896c1b53366555ff (DEFAULT_TABLE table=default.Table_cca695fad03947debe4d1d515b3c73d8 [id=9d441f2151974914945282a0e0c4c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.093191 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 185cc5f8c17146fa92dda7d31cde4416 (DEFAULT_TABLE table=default.Table_7fed58fcd48d4bd6ba7e1ce4d4c40635 [id=089048e00fbc4e7b8b14e185f89df7f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.093250 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f4de297274444982095e9c308f0983. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.093712 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c969377aaf034e9b896c1b53366555ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.094851 1154 tablet_bootstrap.cc:492] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.095085 1154 ts_tablet_manager.cc:1397] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.009s
I20250629 01:51:48.095775 1154 raft_consensus.cc:357] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.095923 1154 raft_consensus.cc:383] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.096019 1154 raft_consensus.cc:738] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.096235 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d5eb1b3c3345c099f298df240a5c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.096262 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4ed143377709465d9ae5df7c2ec3ca47" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.096264 1154 consensus_queue.cc:260] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.096740 1154 ts_tablet_manager.cc:1428] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:48.097021 1154 tablet_bootstrap.cc:492] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.098519 1154 tablet_bootstrap.cc:654] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.100010 880 tablet_service.cc:1468] Processing CreateTablet for tablet 25f4de297274444982095e9c308f0983 (DEFAULT_TABLE table=default.Table_711c32a531fd46489436ed1a683d3297 [id=c78a485212fc4b19a5b17df637acb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:48.100271 988 leader_election.cc:343] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.100396 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f4de297274444982095e9c308f0983. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.100410 988 leader_election.cc:304] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.100788 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185cc5f8c17146fa92dda7d31cde4416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.106544 883 tablet_service.cc:1468] Processing CreateTablet for tablet 07d5eb1b3c3345c099f298df240a5c75 (DEFAULT_TABLE table=default.Table_da7726c4aa4a4bf893ec5b19264cd6ea [id=bc368c1f84884f5d988a9b71a78dd65e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.106886 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07d5eb1b3c3345c099f298df240a5c75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.109346 1154 tablet_bootstrap.cc:492] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.109504 1154 ts_tablet_manager.cc:1397] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20250629 01:51:48.110394 1154 raft_consensus.cc:357] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.110708 1154 raft_consensus.cc:383] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.110843 1154 raft_consensus.cc:738] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.111202 1154 consensus_queue.cc:260] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.112501 1154 ts_tablet_manager.cc:1428] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.112511 885 tablet_service.cc:1468] Processing CreateTablet for tablet c969377aaf034e9b896c1b53366555ff (DEFAULT_TABLE table=default.Table_cca695fad03947debe4d1d515b3c73d8 [id=9d441f2151974914945282a0e0c4c604]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.112839 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c969377aaf034e9b896c1b53366555ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.113008 1154 tablet_bootstrap.cc:492] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.114440 1720 raft_consensus.cc:2747] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.115141 1154 tablet_bootstrap.cc:654] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.118356 891 tablet_service.cc:1468] Processing CreateTablet for tablet d73c2eb0d69e467b94a250e1790d8142 (DEFAULT_TABLE table=default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.118698 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d73c2eb0d69e467b94a250e1790d8142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.124377 885 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf612fea5b74a0bb78700cc07c4d68d (DEFAULT_TABLE table=default.Table_65f2522f26a440dca7e88a9df18ee4e0 [id=3ef8548791d04f849c7a5ebb4233190a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.124676 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf612fea5b74a0bb78700cc07c4d68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.126865 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf612fea5b74a0bb78700cc07c4d68d (DEFAULT_TABLE table=default.Table_65f2522f26a440dca7e88a9df18ee4e0 [id=3ef8548791d04f849c7a5ebb4233190a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.127285 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf612fea5b74a0bb78700cc07c4d68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.128427 1154 tablet_bootstrap.cc:492] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.128634 1154 ts_tablet_manager.cc:1397] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20250629 01:51:48.129524 1154 raft_consensus.cc:357] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.129773 1154 raft_consensus.cc:383] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.129925 1154 raft_consensus.cc:738] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.130278 886 tablet_service.cc:1468] Processing CreateTablet for tablet 12873ec9a5574e789f9d669ddda85704 (DEFAULT_TABLE table=default.Table_6864f50c0e624776b4090fb5368ae3dc [id=8fb3448eefdc46cbbe3f9d8cc52c6c11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.130252 1154 consensus_queue.cc:260] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.130609 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12873ec9a5574e789f9d669ddda85704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.130993 1154 ts_tablet_manager.cc:1428] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:48.131842 1154 tablet_bootstrap.cc:492] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.132676 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1bf612fea5b74a0bb78700cc07c4d68d (DEFAULT_TABLE table=default.Table_65f2522f26a440dca7e88a9df18ee4e0 [id=3ef8548791d04f849c7a5ebb4233190a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.133025 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bf612fea5b74a0bb78700cc07c4d68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.133674 1154 tablet_bootstrap.cc:654] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.136021 884 tablet_service.cc:1468] Processing CreateTablet for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 (DEFAULT_TABLE table=default.Table_6429e6545a1d4d5bb1b37b6799c105ad [id=e87d9a774eb1499f8c9d91bdf13aa08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.136335 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.140058 1154 tablet_bootstrap.cc:492] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.140244 1154 ts_tablet_manager.cc:1397] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20250629 01:51:48.140588 1155 tablet_bootstrap.cc:492] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.140717 1155 ts_tablet_manager.cc:1397] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.296s user 0.081s sys 0.130s
I20250629 01:51:48.141049 880 tablet_service.cc:1468] Processing CreateTablet for tablet 185cc5f8c17146fa92dda7d31cde4416 (DEFAULT_TABLE table=default.Table_7fed58fcd48d4bd6ba7e1ce4d4c40635 [id=089048e00fbc4e7b8b14e185f89df7f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.141125 1154 raft_consensus.cc:357] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.141350 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 185cc5f8c17146fa92dda7d31cde4416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.141381 1154 raft_consensus.cc:383] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.141427 1155 raft_consensus.cc:357] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.141521 1154 raft_consensus.cc:738] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.141583 1155 raft_consensus.cc:383] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.141669 1155 raft_consensus.cc:738] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.141844 1154 consensus_queue.cc:260] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.141860 1155 consensus_queue.cc:260] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.142433 1154 ts_tablet_manager.cc:1428] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:48.142554 1155 ts_tablet_manager.cc:1428] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.143216 1154 tablet_bootstrap.cc:492] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.143502 1016 tablet_service.cc:1468] Processing CreateTablet for tablet d73c2eb0d69e467b94a250e1790d8142 (DEFAULT_TABLE table=default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.143831 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d73c2eb0d69e467b94a250e1790d8142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:47.997557 1159 tablet_bootstrap.cc:654] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.144764 1154 tablet_bootstrap.cc:654] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.150323 744 tablet_service.cc:1468] Processing CreateTablet for tablet d73c2eb0d69e467b94a250e1790d8142 (DEFAULT_TABLE table=default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.156471 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f9574ad3b53b407d9a3352687623467a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.156980 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9574ad3b53b407d9a3352687623467a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.162096 1154 tablet_bootstrap.cc:492] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.162212 1154 ts_tablet_manager.cc:1397] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.002s
I20250629 01:51:48.168283 1155 tablet_bootstrap.cc:492] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.170073 1155 tablet_bootstrap.cc:654] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.162797 1154 raft_consensus.cc:357] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.170773 1154 raft_consensus.cc:383] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.170847 1154 raft_consensus.cc:738] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.170877 1682 tablet.cc:1722] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:48.171008 1154 consensus_queue.cc:260] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.171340 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d73c2eb0d69e467b94a250e1790d8142. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.171464 1154 ts_tablet_manager.cc:1428] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250629 01:51:48.171685 1154 tablet_bootstrap.cc:492] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.172920 1154 tablet_bootstrap.cc:654] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.181115 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 119248ade66640299a7f17fb158a3a5a
I20250629 01:51:48.181746 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ad69c6d3ce44bfd9da5075795ba23a5" } modify_external_catalogs: true
I20250629 01:51:48.181897 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ad69c6d3ce44bfd9da5075795ba23a5" } modify_external_catalogs: true
I20250629 01:51:48.183562 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 689adaf0c7564d9ba0b31061e4e2d8cf
I20250629 01:51:48.184226 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 119248ade66640299a7f17fb158a3a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.184720 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.184953 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 119248ade66640299a7f17fb158a3a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:48.185359 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 119248ade66640299a7f17fb158a3a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 119248ade66640299a7f17fb158a3a5a already in progress: creating tablet
W20250629 01:51:48.185564 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 689adaf0c7564d9ba0b31061e4e2d8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 689adaf0c7564d9ba0b31061e4e2d8cf already in progress: creating tablet
I20250629 01:51:48.185928 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_541d81feb03a418780ca4f7da1ed1d4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.186388 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_541d81feb03a418780ca4f7da1ed1d4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.187453 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.188215 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:51:48.188648 446 catalog_manager.cc:6652] ReplaceTablet: tablet cb7314fbdfc74b1c8d9d1be9c05244cb deleted and replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc
I20250629 01:51:48.188700 915 raft_consensus.cc:3058] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.188889 744 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb with delete_type TABLET_DATA_DELETED (replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.188943 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0400c31cb3764210a6e2492c308e02e8" } new_table_name: "default.Table_99703cee14e34f2ebe6028b072a5b072" modify_external_catalogs: true
W20250629 01:51:48.189384 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet cb7314fbdfc74b1c8d9d1be9c05244cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7314fbdfc74b1c8d9d1be9c05244cb already in progress: creating tablet
I20250629 01:51:48.189985 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b6d21dfd48ea4a95a2fcecde6735974b
I20250629 01:51:48.190601 744 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.191048 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 959837d39a9a4a6f8735a906b3a53222 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:48.191300 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b6d21dfd48ea4a95a2fcecde6735974b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d21dfd48ea4a95a2fcecde6735974b already in progress: creating tablet
W20250629 01:51:48.191956 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 119248ade66640299a7f17fb158a3a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 119248ade66640299a7f17fb158a3a5a already in progress: creating tablet
I20250629 01:51:48.192231 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb with delete_type TABLET_DATA_DELETED (replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:48.192317 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 689adaf0c7564d9ba0b31061e4e2d8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 689adaf0c7564d9ba0b31061e4e2d8cf already in progress: creating tablet
I20250629 01:51:48.192636 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b50cf2cfa18641428019543c0982188d" } new_table_name: "default.Table_0247394d7c3c4941a7ef5ad5fa572566" modify_external_catalogs: true
W20250629 01:51:48.193020 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet cb7314fbdfc74b1c8d9d1be9c05244cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7314fbdfc74b1c8d9d1be9c05244cb already in progress: creating tablet
I20250629 01:51:48.194983 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b0c2dd05f5bb466fbe38ed259ce3a4b8
I20250629 01:51:48.195889 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.196306 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6] alter complete (version 3)
W20250629 01:51:48.196375 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 already in progress: creating tablet
I20250629 01:51:48.196579 744 tablet_service.cc:1515] Processing DeleteTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.196995 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:48.197047 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 already in progress: creating tablet
W20250629 01:51:48.197607 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b6d21dfd48ea4a95a2fcecde6735974b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d21dfd48ea4a95a2fcecde6735974b already in progress: creating tablet
I20250629 01:51:48.198406 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8a60215417c34bedafbf12172eab469a (table default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]). Expected version 1 got 0
I20250629 01:51:48.198709 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:48.199074 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 26d85fd3d3dd4b07a812c019e1a5c748 (table default.Table_4c71f3d9eabc42539dbe4ed7cbf6d738 [id=b9df439e13104b14be7c1b522b04ec13]). Expected version 3 got 0
I20250629 01:51:48.207084 887 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb with delete_type TABLET_DATA_DELETED (replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.207521 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.207952 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.208367 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 36fcb819c61c4fabbb0da295d0002998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.208640 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.208715 450 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]). Expected version 3 got 0
I20250629 01:51:48.208909 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 119248ade66640299a7f17fb158a3a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.210543 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.210739 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a574f95ac2a4193a563e63cbe120768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.210909 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet a32636c293bc44858da7f29a515c467b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.211102 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.211333 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 with delete_type TABLET_DATA_DELETED (replaced by tablet bccdcd9de08648bb925db6fe56f97d20) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.211544 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.211726 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.211885 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.212054 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 881461aa1bc34dd483f65e171c341bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.212258 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.212469 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.212620 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet dd709b557b334434a6c2c4a60a996342 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.212791 1158 tablet_replica.cc:331] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.212961 1158 raft_consensus.cc:2241] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.213102 1158 raft_consensus.cc:2270] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.213800 1158 ts_tablet_manager.cc:1905] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.214293 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 021da4395e414413b9b11ebcfbffcd19
I20250629 01:51:48.216738 1710 raft_consensus.cc:491] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.216929 1710 raft_consensus.cc:513] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.217967 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.218015 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28eaa55b3b564e2e9cf698fda98a4c8a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.218171 1156 tablet_replica.cc:331] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.218353 1156 raft_consensus.cc:2241] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.218508 1156 raft_consensus.cc:2270] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:48.218489 986 leader_election.cc:343] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.219084 1710 leader_election.cc:290] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.219467 1156 ts_tablet_manager.cc:1905] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.219522 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "28eaa55b3b564e2e9cf698fda98a4c8a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.220043 988 leader_election.cc:343] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.220192 988 leader_election.cc:304] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.220652 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c712d978a3848b9bb3abdbce8314dec with delete_type TABLET_DATA_DELETED (replaced by tablet 0323947b135a4308903b1eed1ca4db71) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.220852 1710 raft_consensus.cc:2747] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.224411 744 tablet_service.cc:1515] Processing DeleteTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.224829 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.225214 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.225689 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 021da4395e414413b9b11ebcfbffcd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.228711 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 021da4395e414413b9b11ebcfbffcd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.229207 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 021da4395e414413b9b11ebcfbffcd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.230818 1158 ts_tablet_manager.cc:1918] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.234154 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c9445c5e7b1a4c62af297a2184934b9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.234776 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9445c5e7b1a4c62af297a2184934b9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.236052 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_69f7afd6dbc24708810b9a825012ae62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.236574 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69f7afd6dbc24708810b9a825012ae62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.240164 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_379f342e713042b49f4d21aa5e7b87a7" } new_table_name: "default.Table_78b4b6aea1c144eeb3669ca5d081e0db" modify_external_catalogs: true
I20250629 01:51:48.240549 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b557e22870a142208875b21ca442341b" } new_table_name: "default.Table_2e7ef5453b1945728384798f25e47b7f" modify_external_catalogs: true
I20250629 01:51:48.246332 1664 raft_consensus.cc:491] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.246476 1664 raft_consensus.cc:513] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:48.246614 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bdc6a26b3594d6fb33a27cac5b29f4f consensus state has no leader
I20250629 01:51:48.247017 1664 leader_election.cc:290] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.247166 1664 raft_consensus.cc:491] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.247296 1664 raft_consensus.cc:513] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:48.247403 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbfcb9233ace429db5daa593c2a1dcbe consensus state has no leader
I20250629 01:51:48.247785 1664 leader_election.cc:290] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.248536 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a4e5b66c75942cc8e3e9dfe5a69bf8f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.248699 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "955a41750b6a4766bb4fd3f01cfd99c8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.248991 712 leader_election.cc:343] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:48.249158 712 leader_election.cc:343] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:48.249873 710 leader_election.cc:343] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:48.249998 710 leader_election.cc:304] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:48.250363 1664 raft_consensus.cc:2747] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.252067 743 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.253335 1159 tablet_bootstrap.cc:492] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:48.253475 1159 ts_tablet_manager.cc:1397] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.263s user 0.001s sys 0.001s
I20250629 01:51:48.254127 1159 raft_consensus.cc:357] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.254288 1159 raft_consensus.cc:383] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.254354 1159 raft_consensus.cc:738] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.254776 1159 consensus_queue.cc:260] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.255342 1159 ts_tablet_manager.cc:1428] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.255717 1159 tablet_bootstrap.cc:492] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:48.258584 1156 ts_tablet_manager.cc:1918] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.249339 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "955a41750b6a4766bb4fd3f01cfd99c8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:48.259078 1049 raft_consensus.cc:2466] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:48.259673 1158 log.cc:1199] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/75fd4ffcf896419b81a1c0eb7c5aeeec
I20250629 01:51:48.259501 710 leader_election.cc:304] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:48.260136 1664 raft_consensus.cc:2802] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:48.260216 1664 raft_consensus.cc:491] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:48.260305 1664 raft_consensus.cc:3058] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.262569 1664 raft_consensus.cc:513] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.263545 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "955a41750b6a4766bb4fd3f01cfd99c8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:48.263667 1664 leader_election.cc:290] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:48.264029 712 leader_election.cc:343] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.264297 1156 log.cc:1199] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5dfe921a84694903bbdb3fed3d6f2761
I20250629 01:51:48.264623 1156 ts_tablet_manager.cc:1939] T 5dfe921a84694903bbdb3fed3d6f2761 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.265897 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5dfe921a84694903bbdb3fed3d6f2761 (table default.Table_6ada0762e0224924acf20a4507ad2fa6 [id=82f79023b3c0425c978152033e2183b3]) successfully deleted
I20250629 01:51:48.264009 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "955a41750b6a4766bb4fd3f01cfd99c8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:48.266222 1049 raft_consensus.cc:3058] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.270120 1049 raft_consensus.cc:2466] T 955a41750b6a4766bb4fd3f01cfd99c8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:48.270488 710 leader_election.cc:304] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:48.270844 1664 raft_consensus.cc:2802] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:48.270968 1664 raft_consensus.cc:695] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:48.271198 1664 consensus_queue.cc:237] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.276968 885 tablet_service.cc:1468] Processing CreateTablet for tablet a2e9de48e4ad401880f351f2bd55cf5c (DEFAULT_TABLE table=default.Table_d01934d1cb70450dbb1b6cf0c40c2129 [id=937ad8d7d20745ab857b9698a18602b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.277208 743 tablet_service.cc:1468] Processing CreateTablet for tablet a2e9de48e4ad401880f351f2bd55cf5c (DEFAULT_TABLE table=default.Table_d01934d1cb70450dbb1b6cf0c40c2129 [id=937ad8d7d20745ab857b9698a18602b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.277580 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e9de48e4ad401880f351f2bd55cf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.278013 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e9de48e4ad401880f351f2bd55cf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.279495 1710 raft_consensus.cc:491] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.279618 1710 raft_consensus.cc:513] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.280345 1710 leader_election.cc:290] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.280361 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5fba3653ae2c4de78346d2983120e610"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.280586 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4080248559414e1b92597c0367d2530c" } new_table_name: "default.Table_1912093cd0fd4042acd742fa698b5c83" modify_external_catalogs: true
I20250629 01:51:48.280740 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a32636c293bc44858da7f29a515c467b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:48.280817 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fba3653ae2c4de78346d2983120e610 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:48.281188 986 leader_election.cc:343] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.281493 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c2146a78ba3746aea6dc9d3614b1e5c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.281932 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2146a78ba3746aea6dc9d3614b1e5c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.282099 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f8e69accdc0a4e35a1d60c963e638c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.282516 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8e69accdc0a4e35a1d60c963e638c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.283581 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a32636c293bc44858da7f29a515c467b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.283978 988 leader_election.cc:343] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.284113 988 leader_election.cc:304] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.284663 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.285065 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.285909 1710 raft_consensus.cc:2747] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.286304 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a2e9de48e4ad401880f351f2bd55cf5c (DEFAULT_TABLE table=default.Table_d01934d1cb70450dbb1b6cf0c40c2129 [id=937ad8d7d20745ab857b9698a18602b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.286687 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e9de48e4ad401880f351f2bd55cf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.287248 1155 tablet_bootstrap.cc:492] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.287418 1155 ts_tablet_manager.cc:1397] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.119s user 0.025s sys 0.025s
I20250629 01:51:48.288087 1155 raft_consensus.cc:357] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.288290 1155 raft_consensus.cc:383] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.288365 1155 raft_consensus.cc:738] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.288573 1155 consensus_queue.cc:260] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.289175 1159 tablet_bootstrap.cc:654] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.289459 1154 tablet_bootstrap.cc:492] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.289614 1154 ts_tablet_manager.cc:1397] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.118s user 0.018s sys 0.031s
I20250629 01:51:48.290257 1154 raft_consensus.cc:357] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.290478 1154 raft_consensus.cc:383] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.290568 1154 raft_consensus.cc:738] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.290783 1154 consensus_queue.cc:260] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.292552 1154 ts_tablet_manager.cc:1428] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:48.292837 1154 tablet_bootstrap.cc:492] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:48.293228 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce90c045115844b0b96143f91f1be1d4 consensus state has no leader
I20250629 01:51:48.293704 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:48.293876 1158 ts_tablet_manager.cc:1939] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.294186 1154 tablet_bootstrap.cc:654] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.294387 1710 raft_consensus.cc:491] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.294481 1710 raft_consensus.cc:513] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.295356 886 tablet_service.cc:1468] Processing CreateTablet for tablet 8643745db0954c248e915ee1dbf8ac0e (DEFAULT_TABLE table=default.Table_37e5f93dba384c5c930e4028ae2a8110 [id=7c6ec02ad8ea4320a8268c853a9a9679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.295727 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 75fd4ffcf896419b81a1c0eb7c5aeeec (table default.Table_5ce4c958f3d4456fb56450ca4cc26221 [id=646791dd9bb447fdb2e96a9046dd68d2]) successfully deleted
I20250629 01:51:48.295091 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 8643745db0954c248e915ee1dbf8ac0e (DEFAULT_TABLE table=default.Table_37e5f93dba384c5c930e4028ae2a8110 [id=7c6ec02ad8ea4320a8268c853a9a9679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.296286 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8643745db0954c248e915ee1dbf8ac0e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.297288 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da93c730451d42bc8df1e582fb3af818 consensus state has no leader
W20250629 01:51:48.297581 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47cbc958eca14ed69fb45c3c2984aeef consensus state has no leader
I20250629 01:51:48.297608 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:48.300382 743 tablet_service.cc:1468] Processing CreateTablet for tablet 8643745db0954c248e915ee1dbf8ac0e (DEFAULT_TABLE table=default.Table_37e5f93dba384c5c930e4028ae2a8110 [id=7c6ec02ad8ea4320a8268c853a9a9679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.300762 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8643745db0954c248e915ee1dbf8ac0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.301955 1710 leader_election.cc:290] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.302333 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd709b557b334434a6c2c4a60a996342" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:48.302774 986 leader_election.cc:343] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.303239 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd709b557b334434a6c2c4a60a996342" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.303912 988 leader_election.cc:343] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.304016 988 leader_election.cc:304] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.305886 1710 raft_consensus.cc:2747] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.306224 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8643745db0954c248e915ee1dbf8ac0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.310212 884 tablet_service.cc:1468] Processing CreateTablet for tablet b75d33594c994363bdb2558eb0c6ca3d (DEFAULT_TABLE table=default.Table_82d6223100ea4aba8941923226f2baa6 [id=5f7e5341feb144b3b68e75b92c888d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.310598 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b75d33594c994363bdb2558eb0c6ca3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.312486 1158 tablet_replica.cc:331] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.312709 1158 raft_consensus.cc:2241] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.312914 1158 raft_consensus.cc:2270] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.313548 1158 ts_tablet_manager.cc:1905] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.314097 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b75d33594c994363bdb2558eb0c6ca3d (DEFAULT_TABLE table=default.Table_82d6223100ea4aba8941923226f2baa6 [id=5f7e5341feb144b3b68e75b92c888d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.314308 1157 ts_tablet_manager.cc:1918] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.314450 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b75d33594c994363bdb2558eb0c6ca3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.314579 1710 raft_consensus.cc:491] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.314728 1710 raft_consensus.cc:513] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.315984 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d85fd3d3dd4b07a812c019e1a5c748" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:48.316455 986 leader_election.cc:343] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.316638 1710 leader_election.cc:290] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.319253 1158 ts_tablet_manager.cc:1918] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.320483 1156 tablet_replica.cc:331] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.320662 1156 raft_consensus.cc:2241] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.320806 1156 raft_consensus.cc:2270] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.323609 1156 ts_tablet_manager.cc:1905] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.325788 1158 log.cc:1199] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0a574f95ac2a4193a563e63cbe120768
I20250629 01:51:48.326591 747 tablet_service.cc:1468] Processing CreateTablet for tablet b75d33594c994363bdb2558eb0c6ca3d (DEFAULT_TABLE table=default.Table_82d6223100ea4aba8941923226f2baa6 [id=5f7e5341feb144b3b68e75b92c888d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.327153 1158 ts_tablet_manager.cc:1939] T 0a574f95ac2a4193a563e63cbe120768 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.327366 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b75d33594c994363bdb2558eb0c6ca3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.328511 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0a574f95ac2a4193a563e63cbe120768 (table default.Table_399e960d59fe4143843457bd372dbf63 [id=99712a47e4424c78bb4721eeec6e90c3]) successfully deleted
I20250629 01:51:48.330077 1158 tablet_replica.cc:331] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.330629 1158 raft_consensus.cc:2241] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.331279 1158 raft_consensus.cc:2270] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.332486 1158 ts_tablet_manager.cc:1905] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.333876 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a2562570cb0846e19ebc97bdb43bd70d (table default.Table_f07aa6eb42b6478781eb5a65dea53212 [id=9ff841a004c344c688f5abf6b59b5dce]). Expected version 6 got 0
I20250629 01:51:48.343456 1158 ts_tablet_manager.cc:1918] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.345264 1158 log.cc:1199] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a32636c293bc44858da7f29a515c467b
I20250629 01:51:48.345728 1158 ts_tablet_manager.cc:1939] T a32636c293bc44858da7f29a515c467b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.346783 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a32636c293bc44858da7f29a515c467b (table default.Table_2fc84dd48d464ce682425fbd41d92b7e [id=64b627ee531b41ffa46d4bbd813c5b8c]) successfully deleted
I20250629 01:51:48.347628 1158 tablet_replica.cc:331] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.347985 1158 raft_consensus.cc:2241] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.348348 1158 raft_consensus.cc:2270] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.349221 1158 ts_tablet_manager.cc:1905] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.350968 1156 ts_tablet_manager.cc:1918] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.351099 1156 log.cc:1199] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/af9ab324ce7c488cbaf6e0f0322b5c58
I20250629 01:51:48.351482 1156 ts_tablet_manager.cc:1939] T af9ab324ce7c488cbaf6e0f0322b5c58 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.351428 451 catalog_manager.cc:5582] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:48.356598 1154 tablet_bootstrap.cc:492] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.356740 1710 raft_consensus.cc:491] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.356905 1710 raft_consensus.cc:513] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.357658 1710 leader_election.cc:290] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.363976 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 8295e8178eff4d5eb95ac885092731e0 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.365185 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet af9ab324ce7c488cbaf6e0f0322b5c58 (table default.Table_2c04531a50c54791a64d6537b670d54f [id=e4025f53c2ea4b7aa8fafb057c102a3b]) successfully deleted
I20250629 01:51:48.369730 1158 ts_tablet_manager.cc:1918] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.369832 1158 log.cc:1199] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/be8e0946a20340b38835c4d8daeaa7b8
I20250629 01:51:48.370095 1158 ts_tablet_manager.cc:1939] T be8e0946a20340b38835c4d8daeaa7b8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.371780 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet be8e0946a20340b38835c4d8daeaa7b8 (table default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]) successfully deleted
I20250629 01:51:48.372131 1156 tablet_replica.cc:331] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.372469 1156 raft_consensus.cc:2241] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.372694 1156 raft_consensus.cc:2270] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.373188 1156 ts_tablet_manager.cc:1905] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.375193 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 2249d5cba1544653b088312e7d74802e from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.376755 1158 tablet_replica.cc:331] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.377172 1158 raft_consensus.cc:2241] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.377315 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a60215417c34bedafbf12172eab469a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:48.377810 986 leader_election.cc:343] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.377939 1158 raft_consensus.cc:2270] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.378016 1766 raft_consensus.cc:491] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.378717 1158 ts_tablet_manager.cc:1905] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.356739 1154 ts_tablet_manager.cc:1397] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.009s
I20250629 01:51:48.380803 1154 raft_consensus.cc:357] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.380949 1154 raft_consensus.cc:383] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.381028 1154 raft_consensus.cc:738] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.381215 1154 consensus_queue.cc:260] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.381680 1154 ts_tablet_manager.cc:1428] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:48.383770 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 2049e7465da4404a86dbaf5a77adcda6 (DEFAULT_TABLE table=default.Table_3821b9a8d745433382baadfd96b6c260 [id=70a505857d794534a8699de74c1a7614]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.384027 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2049e7465da4404a86dbaf5a77adcda6 (DEFAULT_TABLE table=default.Table_3821b9a8d745433382baadfd96b6c260 [id=70a505857d794534a8699de74c1a7614]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.384433 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2049e7465da4404a86dbaf5a77adcda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.384974 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.385017 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1f1cf7aea79d46d9998b288e0c782dd6" } modify_external_catalogs: true
I20250629 01:51:48.385166 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1f1cf7aea79d46d9998b288e0c782dd6" } modify_external_catalogs: true
I20250629 01:51:48.386137 1158 ts_tablet_manager.cc:1918] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.386242 1158 log.cc:1199] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a90b206d58d04acdb1a8cb0c45d68c31
I20250629 01:51:48.386327 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8b976eb77218429b9513d370b45cd5c9" } modify_external_catalogs: true
I20250629 01:51:48.386461 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8b976eb77218429b9513d370b45cd5c9" } modify_external_catalogs: true
I20250629 01:51:48.386529 1158 ts_tablet_manager.cc:1939] T a90b206d58d04acdb1a8cb0c45d68c31 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.388149 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e3874404c3444d47a71f1dce7fb86ff8 (DEFAULT_TABLE table=default.Table_ea4ec96134d342ad8386604b1305651d [id=40849cc1f1d74e0ebc9cd6a2f117dbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.388419 1155 ts_tablet_manager.cc:1428] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.101s user 0.008s sys 0.019s
I20250629 01:51:48.388738 1155 tablet_bootstrap.cc:492] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.378177 1766 raft_consensus.cc:513] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.389380 747 tablet_service.cc:1468] Processing CreateTablet for tablet e3874404c3444d47a71f1dce7fb86ff8 (DEFAULT_TABLE table=default.Table_ea4ec96134d342ad8386604b1305651d [id=40849cc1f1d74e0ebc9cd6a2f117dbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.390367 1155 tablet_bootstrap.cc:654] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.390702 1766 leader_election.cc:290] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.392151 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 28eaa55b3b564e2e9cf698fda98a4c8a
I20250629 01:51:48.392532 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3874404c3444d47a71f1dce7fb86ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.392800 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3070ce8c467a47b0a833374450911291
I20250629 01:51:48.398294 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8c1fbbaabe004466858cce35b18194b1" } modify_external_catalogs: true
I20250629 01:51:48.398453 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8c1fbbaabe004466858cce35b18194b1" } modify_external_catalogs: true
I20250629 01:51:48.400614 1155 tablet_bootstrap.cc:492] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.400777 1155 ts_tablet_manager.cc:1397] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20250629 01:51:48.400774 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2c656b7d46064f7d811de6d03aa4224e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.400970 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5468f28974964427aa77e0d4809ca02c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.401257 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c656b7d46064f7d811de6d03aa4224e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:48.401355 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5468f28974964427aa77e0d4809ca02c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.401472 1155 raft_consensus.cc:357] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.401604 1155 raft_consensus.cc:383] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.401679 1155 raft_consensus.cc:738] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.401844 1155 consensus_queue.cc:260] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.402227 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_35d223e2a4d24c85a2a3780651c205e2" } modify_external_catalogs: true
I20250629 01:51:48.402276 1155 ts_tablet_manager.cc:1428] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:48.402369 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_35d223e2a4d24c85a2a3780651c205e2" } modify_external_catalogs: true
I20250629 01:51:48.402514 1155 tablet_bootstrap.cc:492] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.404451 1155 tablet_bootstrap.cc:654] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.406776 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2249d5cba1544653b088312e7d74802e
I20250629 01:51:48.407182 451 catalog_manager.cc:6652] ReplaceTablet: tablet 2249d5cba1544653b088312e7d74802e deleted and replaced by tablet d387af53209147a6bac7afeda9a411ab
I20250629 01:51:48.411463 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.412494 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3070ce8c467a47b0a833374450911291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.413338 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 2249d5cba1544653b088312e7d74802e with delete_type TABLET_DATA_DELETED (replaced by tablet d387af53209147a6bac7afeda9a411ab) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.415482 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_44f5d1913b0e4eedaac117acfdf02c22" } modify_external_catalogs: true
I20250629 01:51:48.415628 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_44f5d1913b0e4eedaac117acfdf02c22" } modify_external_catalogs: true
I20250629 01:51:48.417014 455 master_service.cc:948] ReplaceTablet: received request to replace tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.419651 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0665d3f1225c44f894681fef4ae2520b
I20250629 01:51:48.423306 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 0665d3f1225c44f894681fef4ae2520b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.423498 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0827ec622ff04ca6b2fec50e40b9e1de
I20250629 01:51:48.423581 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 0665d3f1225c44f894681fef4ae2520b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.423923 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f3bdc8bda8a940be865679a05b90bd08
I20250629 01:51:48.404839 1156 ts_tablet_manager.cc:1918] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.425061 1156 log.cc:1199] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d663e7dc7a384bf6bd0cbfa9d9e5e9f0
I20250629 01:51:48.425086 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f3bdc8bda8a940be865679a05b90bd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.425280 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f3bdc8bda8a940be865679a05b90bd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.413254 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 2249d5cba1544653b088312e7d74802e with delete_type TABLET_DATA_DELETED (replaced by tablet d387af53209147a6bac7afeda9a411ab) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.413767 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7a640451934bff8672cd88dc86c529 with delete_type TABLET_DATA_DELETED (replaced by tablet d456410cd09845dabd979c123f70741b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.425513 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.425609 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3874404c3444d47a71f1dce7fb86ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.425585 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:48.426012 710 leader_election.cc:343] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:48.416205 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 3070ce8c467a47b0a833374450911291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.427031 1154 tablet_bootstrap.cc:492] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.427191 1156 ts_tablet_manager.cc:1939] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.428148 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 (table default.Table_737eb1cf8b184a14b253c25d114a14b7 [id=eda52aa556274b26af9896e638b51e7a]) successfully deleted
I20250629 01:51:48.428339 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2049e7465da4404a86dbaf5a77adcda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.428429 1156 tablet_replica.cc:331] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.428632 1156 raft_consensus.cc:2241] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.428864 1156 raft_consensus.cc:2270] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.429339 1156 ts_tablet_manager.cc:1905] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.429418 1154 tablet_bootstrap.cc:654] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.429790 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 351f4a7ae7e244999dda81d282de505d with delete_type TABLET_DATA_DELETED (replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.430682 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.317004 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d85fd3d3dd4b07a812c019e1a5c748" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.319010 1690 tablet.cc:1722] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20250629 01:51:48.336814 883 tablet_service.cc:1515] Processing DeleteTablet for tablet af9ab324ce7c488cbaf6e0f0322b5c58 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.358098 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a60215417c34bedafbf12172eab469a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.372730 1159 tablet_bootstrap.cc:492] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:48.432389 1159 ts_tablet_manager.cc:1397] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.177s user 0.003s sys 0.000s
I20250629 01:51:48.432502 1157 log.cc:1199] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a2a9664bb7a946c48cf79a09955ecad5
I20250629 01:51:48.432891 1157 ts_tablet_manager.cc:1939] T a2a9664bb7a946c48cf79a09955ecad5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:48.433194 1159 raft_consensus.cc:357] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.433370 1159 raft_consensus.cc:383] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.433451 1159 raft_consensus.cc:738] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.433637 1159 consensus_queue.cc:260] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.434089 1159 ts_tablet_manager.cc:1428] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:48.434367 1159 tablet_bootstrap.cc:492] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:48.437430 988 leader_election.cc:343] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.437541 988 leader_election.cc:304] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.437885 1710 raft_consensus.cc:2747] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.387943 891 tablet_service.cc:1468] Processing CreateTablet for tablet 2049e7465da4404a86dbaf5a77adcda6 (DEFAULT_TABLE table=default.Table_3821b9a8d745433382baadfd96b6c260 [id=70a505857d794534a8699de74c1a7614]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.438354 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2049e7465da4404a86dbaf5a77adcda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.439225 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_59e6d6754ff145b8bea3842035e15189"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.439632 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59e6d6754ff145b8bea3842035e15189 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.390568 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bcc3f2a76bc4bae8b306b54e75656cc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.443430 914 raft_consensus.cc:2466] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:48.392844 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.395601 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 3070ce8c467a47b0a833374450911291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.420367 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 0665d3f1225c44f894681fef4ae2520b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.422676 892 tablet_service.cc:1468] Processing CreateTablet for tablet e3874404c3444d47a71f1dce7fb86ff8 (DEFAULT_TABLE table=default.Table_ea4ec96134d342ad8386604b1305651d [id=40849cc1f1d74e0ebc9cd6a2f117dbfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.426779 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.426918 888 tablet_service.cc:1515] Processing DeleteTablet for tablet f3bdc8bda8a940be865679a05b90bd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.444360 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3874404c3444d47a71f1dce7fb86ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.447815 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_27e248a6eaa04f5d8039135e8ff089e1" } new_table_name: "default.Table_b1e075cbb0dc4e5eb8d4de302ab08610" modify_external_catalogs: true
I20250629 01:51:48.448515 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d2c40cf88602426bbe61a1718e57da6c" } new_table_name: "default.Table_3fc2c8b5061f4a6e8f46cd439cf5c776" modify_external_catalogs: true
I20250629 01:51:48.449646 1689 raft_consensus.cc:491] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.449752 1689 raft_consensus.cc:513] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.450280 1689 leader_election.cc:290] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:48.450738 988 leader_election.cc:343] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.450841 988 leader_election.cc:304] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.451196 1710 raft_consensus.cc:2747] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:48.452708 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a792a26725954adea15ad57e31bfb224 consensus state has no leader
W20250629 01:51:48.452700 850 leader_election.cc:343] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 36fcb819c61c4fabbb0da295d0002998
I20250629 01:51:48.455080 1159 tablet_bootstrap.cc:654] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:48.456182 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f94d8fc1b6254ba69acadcaff3613d97 consensus state has no leader
I20250629 01:51:48.459241 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f4cdf270acad4465adbb179a26cedb39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.459892 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4cdf270acad4465adbb179a26cedb39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.437225 1155 tablet_bootstrap.cc:492] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.464449 1155 ts_tablet_manager.cc:1397] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.004s
I20250629 01:51:48.465102 1155 raft_consensus.cc:357] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.465273 1155 raft_consensus.cc:383] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.465335 1155 raft_consensus.cc:738] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.465524 1155 consensus_queue.cc:260] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.465960 1155 ts_tablet_manager.cc:1428] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:48.466233 1155 tablet_bootstrap.cc:492] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.467614 1155 tablet_bootstrap.cc:654] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.443768 712 leader_election.cc:304] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:48.444659 1156 ts_tablet_manager.cc:1918] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.468683 1664 raft_consensus.cc:2802] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:48.468765 1664 raft_consensus.cc:491] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:48.468856 1664 raft_consensus.cc:3058] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.471323 1664 raft_consensus.cc:513] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.472443 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bcc3f2a76bc4bae8b306b54e75656cc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:48.472601 914 raft_consensus.cc:3058] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.472942 1664 leader_election.cc:290] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.473508 1156 log.cc:1199] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:48.473829 1156 ts_tablet_manager.cc:1939] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.474704 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f (table default.Table_5f4d9b39ee1546eca5a1fb0701b2b93f [id=d5edb2619f57401898104c5d2d036b33]) successfully deleted
W20250629 01:51:48.475339 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 021da4395e414413b9b11ebcfbffcd19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 021da4395e414413b9b11ebcfbffcd19 already in progress: creating tablet
I20250629 01:51:48.475631 1769 raft_consensus.cc:491] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.475731 1769 raft_consensus.cc:513] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.475953 914 raft_consensus.cc:2466] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:48.476260 1769 leader_election.cc:290] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:48.476297 850 leader_election.cc:343] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 36fcb819c61c4fabbb0da295d0002998
I20250629 01:51:48.476413 850 leader_election.cc:304] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.476732 1689 raft_consensus.cc:2747] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:48.476789 710 leader_election.cc:343] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:48.477088 712 leader_election.cc:304] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:48.477344 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b7c7f2ac44734a39ab6434a74c91b9e4" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.477427 710 leader_election.cc:343] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b7c7f2ac44734a39ab6434a74c91b9e4
I20250629 01:51:48.477684 1664 raft_consensus.cc:2802] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
W20250629 01:51:48.477872 712 leader_election.cc:343] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.477993 712 leader_election.cc:304] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:48.478317 1769 raft_consensus.cc:2747] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.477880 1664 raft_consensus.cc:695] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:48.478834 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_620b12abd6f9476e8bf5127f618d9295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.479048 1664 consensus_queue.cc:237] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:48.479436 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_620b12abd6f9476e8bf5127f618d9295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.485092 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.485430 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_13dc4660837b46219b404433b7272fce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.485879 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13dc4660837b46219b404433b7272fce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.486282 1156 tablet_replica.cc:331] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.486783 1156 raft_consensus.cc:2241] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.487303 1156 raft_consensus.cc:2270] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.488277 1156 ts_tablet_manager.cc:1905] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.494451 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 2249d5cba1544653b088312e7d74802e with delete_type TABLET_DATA_DELETED (replaced by tablet d387af53209147a6bac7afeda9a411ab) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.495788 1158 tablet_replica.cc:331] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.495843 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a90b206d58d04acdb1a8cb0c45d68c31 (table default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]) successfully deleted
I20250629 01:51:48.495977 1158 raft_consensus.cc:2241] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.496052 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4c71f3d9eabc42539dbe4ed7cbf6d738" } new_table_name: "default.Table_331d3b611f4f46748132e37fb1a8e872" modify_external_catalogs: true
I20250629 01:51:48.496176 1158 raft_consensus.cc:2270] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.498425 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6864f50c0e624776b4090fb5368ae3dc" } new_table_name: "default.Table_6794618e8690495596d2c46dc14125b2" modify_external_catalogs: true
I20250629 01:51:48.498904 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_93358968e4994116ad323cf7bdd275d3" } new_table_name: "default.Table_1602ef35993a40909b316edb27f35b40" modify_external_catalogs: true
I20250629 01:51:48.500069 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bcc3f2a76bc4bae8b306b54e75656cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.500648 1158 ts_tablet_manager.cc:1905] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:48.503475 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12873ec9a5574e789f9d669ddda85704 consensus state has no leader
W20250629 01:51:48.504817 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 556c2050d2cb40f28a3a0bfb9d677d69 consensus state has no leader
I20250629 01:51:48.433215 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 544) took 1606 ms. Trace:
I20250629 01:51:48.506024 1157 rpcz_store.cc:276] 0629 01:51:46.826516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.826659 (+ 143us) service_pool.cc:225] Handling call
0629 01:51:48.050857 (+1224198us) ts_tablet_manager.cc:1145] Deleting tablet a2a9664bb7a946c48cf79a09955ecad5
0629 01:51:48.050864 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.054980 (+ 4116us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:48.314207 (+259227us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:48.433183 (+118976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203431}
W20250629 01:51:48.507963 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 26d85fd3d3dd4b07a812c019e1a5c748 consensus state has no leader
I20250629 01:51:48.508539 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a2a9664bb7a946c48cf79a09955ecad5 (table default.Table_ca2bcac172414e6c9f264cb2202a76fa [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:48.513319 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eae63233f14743f6ba4f11140b92f156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.513713 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eae63233f14743f6ba4f11140b92f156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.515939 1156 ts_tablet_manager.cc:1918] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.517079 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_217838e5ec6344b8bfad7c93057cf5a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.517553 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_217838e5ec6344b8bfad7c93057cf5a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.518057 1156 log.cc:1199] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b7c7f2ac44734a39ab6434a74c91b9e4
I20250629 01:51:48.518119 1710 raft_consensus.cc:491] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.518247 1710 raft_consensus.cc:513] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.518671 1156 ts_tablet_manager.cc:1939] T b7c7f2ac44734a39ab6434a74c91b9e4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.518875 1710 leader_election.cc:290] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.519446 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.519764 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.520155 986 leader_election.cc:343] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:48.520309 988 leader_election.cc:343] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.520457 988 leader_election.cc:304] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.520812 1710 raft_consensus.cc:2747] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.520932 1158 ts_tablet_manager.cc:1918] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.521096 1158 log.cc:1199] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:48.521169 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b7c7f2ac44734a39ab6434a74c91b9e4 (table default.Table_fa24ba8423a34245b118894025a8b4f4 [id=a07cbcbb14984c538bdd7d1441d52055]) successfully deleted
I20250629 01:51:48.521584 1158 ts_tablet_manager.cc:1939] T d91fa0f049904a1fb5cbd38d1fd1693a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.522349 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d91fa0f049904a1fb5cbd38d1fd1693a (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]) successfully deleted
I20250629 01:51:48.522935 1158 tablet_replica.cc:331] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.523142 1158 raft_consensus.cc:2241] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.523308 1158 raft_consensus.cc:2270] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.523763 1158 ts_tablet_manager.cc:1905] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.524708 1156 tablet_replica.cc:331] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.524878 1156 raft_consensus.cc:2241] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:48.525099 1156 pending_rounds.cc:70] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:48.525158 1156 pending_rounds.cc:77] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759177131704320 op_type: NO_OP noop_request { }
I20250629 01:51:48.525260 1156 raft_consensus.cc:2887] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:48.525352 1156 raft_consensus.cc:2270] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.526952 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f7db52cd26e34c54b5fa3ba338525f24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.527113 1156 ts_tablet_manager.cc:1905] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:48.527530 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7db52cd26e34c54b5fa3ba338525f24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.529834 1158 ts_tablet_manager.cc:1918] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.529983 1158 log.cc:1199] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4ed143377709465d9ae5df7c2ec3ca47
I20250629 01:51:48.530326 1158 ts_tablet_manager.cc:1939] T 4ed143377709465d9ae5df7c2ec3ca47 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.531352 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4ed143377709465d9ae5df7c2ec3ca47 (table default.Table_ace97daf931b45ac9676c39144d2931f [id=1669d2f9d2b842bfb8fdbc03b965669a]) successfully deleted
I20250629 01:51:48.531653 1158 tablet_replica.cc:331] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.531843 1158 raft_consensus.cc:2241] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:48.532083 1158 pending_rounds.cc:70] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:48.532202 1158 pending_rounds.cc:77] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759175292850176 op_type: NO_OP noop_request { }
I20250629 01:51:48.532370 1158 raft_consensus.cc:2887] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:48.532495 1158 raft_consensus.cc:2270] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.533310 1158 ts_tablet_manager.cc:1905] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.533363 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e9c328545beb404bb1a6535d809fd023"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.533905 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9c328545beb404bb1a6535d809fd023 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.534993 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 545) took 1677 ms. Trace:
I20250629 01:51:48.535624 1157 rpcz_store.cc:276] 0629 01:51:46.857023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:46.857072 (+ 49us) service_pool.cc:225] Handling call
0629 01:51:48.534857 (+1677785us) ts_tablet_manager.cc:1145] Deleting tablet f465d0bb46674763abfc8d93f45cd13d
0629 01:51:48.534866 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.534955 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1671188}
W20250629 01:51:48.538184 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f465d0bb46674763abfc8d93f45cd13d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f465d0bb46674763abfc8d93f45cd13d already in progress: creating tablet
I20250629 01:51:48.538620 1158 ts_tablet_manager.cc:1918] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:48.543730 1156 ts_tablet_manager.cc:1918] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:48.544211 1158 log.cc:1199] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5dfe921a84694903bbdb3fed3d6f2761
I20250629 01:51:48.544682 1158 ts_tablet_manager.cc:1939] T 5dfe921a84694903bbdb3fed3d6f2761 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.544188 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 555) took 1531 ms. Trace:
I20250629 01:51:48.543809 1154 tablet_bootstrap.cc:492] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.546563 1154 ts_tablet_manager.cc:1397] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.120s user 0.004s sys 0.008s
I20250629 01:51:48.546530 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5dfe921a84694903bbdb3fed3d6f2761 (table default.Table_6ada0762e0224924acf20a4507ad2fa6 [id=82f79023b3c0425c978152033e2183b3]) successfully deleted
I20250629 01:51:48.547212 1154 raft_consensus.cc:357] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.547412 1154 raft_consensus.cc:383] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.547494 1154 raft_consensus.cc:738] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.547703 1154 consensus_queue.cc:260] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.548198 1154 ts_tablet_manager.cc:1428] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:48.548457 1154 tablet_bootstrap.cc:492] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.548147 1157 rpcz_store.cc:276] 0629 01:51:47.012760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.012848 (+ 88us) service_pool.cc:225] Handling call
0629 01:51:48.538480 (+1525632us) ts_tablet_manager.cc:1145] Deleting tablet 0d4320f615464fedbcab25137952f684
0629 01:51:48.538487 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.544158 (+ 5671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516432}
W20250629 01:51:48.548897 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0d4320f615464fedbcab25137952f684 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4320f615464fedbcab25137952f684 already in progress: creating tablet
I20250629 01:51:48.550329 1154 tablet_bootstrap.cc:654] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.550423 1157 tablet_replica.cc:331] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:48.552253 1158 tablet_replica.cc:331] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.552395 1157 raft_consensus.cc:2241] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.552469 1158 raft_consensus.cc:2241] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.552634 1157 raft_consensus.cc:2270] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.552670 1158 raft_consensus.cc:2270] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.553092 1158 ts_tablet_manager.cc:1905] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.553485 1155 tablet_bootstrap.cc:492] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.554054 1157 ts_tablet_manager.cc:1905] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.554050 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.554204 1155 ts_tablet_manager.cc:1397] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.010s
I20250629 01:51:48.554929 1155 raft_consensus.cc:357] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.555131 1155 raft_consensus.cc:383] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.555230 1155 raft_consensus.cc:738] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.555433 1155 consensus_queue.cc:260] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.558087 1158 ts_tablet_manager.cc:1918] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.558194 1158 log.cc:1199] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/881461aa1bc34dd483f65e171c341bc5
I20250629 01:51:48.558494 1158 ts_tablet_manager.cc:1939] T 881461aa1bc34dd483f65e171c341bc5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.558892 1155 ts_tablet_manager.cc:1428] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:48.559219 1158 tablet_replica.cc:331] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.559307 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 881461aa1bc34dd483f65e171c341bc5 (table default.Table_16db51abc1404d7fa67f1c00a70f9166 [id=a922c0f8e123451fb4060f61472e8d41]) successfully deleted
I20250629 01:51:48.559381 1158 raft_consensus.cc:2241] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.559496 1158 raft_consensus.cc:2270] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.559567 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.559891 1158 ts_tablet_manager.cc:1905] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.561275 1155 tablet_bootstrap.cc:492] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.562783 1155 tablet_bootstrap.cc:654] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.563827 1156 log.cc:1199] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:48.564251 1156 ts_tablet_manager.cc:1939] T 8bcc3f2a76bc4bae8b306b54e75656cc P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.565053 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8bcc3f2a76bc4bae8b306b54e75656cc (table default.Table_f6f27d5499af49359886879547b0ac31 [id=e1d26b8e31044413af1d4bddc58816e7]) successfully deleted
W20250629 01:51:48.565598 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0665d3f1225c44f894681fef4ae2520b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0665d3f1225c44f894681fef4ae2520b already in progress: creating tablet
W20250629 01:51:48.570487 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f3bdc8bda8a940be865679a05b90bd08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bdc8bda8a940be865679a05b90bd08 already in progress: creating tablet
W20250629 01:51:48.571157 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2249d5cba1544653b088312e7d74802e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2249d5cba1544653b088312e7d74802e already in progress: creating tablet
I20250629 01:51:48.571470 1156 tablet_replica.cc:331] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.571666 1156 raft_consensus.cc:2241] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.571858 1156 raft_consensus.cc:2270] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.572587 1156 ts_tablet_manager.cc:1905] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.572832 1154 tablet_bootstrap.cc:492] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.572964 1154 ts_tablet_manager.cc:1397] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250629 01:51:48.573647 1154 raft_consensus.cc:357] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.573793 1154 raft_consensus.cc:383] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.573882 1154 raft_consensus.cc:738] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.574133 1154 consensus_queue.cc:260] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.574599 1154 ts_tablet_manager.cc:1428] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.574887 1154 tablet_bootstrap.cc:492] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.576180 1154 tablet_bootstrap.cc:654] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.576231 1710 raft_consensus.cc:491] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.576336 1710 raft_consensus.cc:513] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.577030 1710 leader_election.cc:290] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:48.577651 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49333302a27f4ac3b76a1e492b3c1b60" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:48.578070 986 leader_election.cc:343] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.578598 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49333302a27f4ac3b76a1e492b3c1b60" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.580279 988 leader_election.cc:343] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.580375 988 leader_election.cc:304] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.580794 1710 raft_consensus.cc:2747] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.581190 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3c70f54de70b4d0e94d25239b097f8fa (DEFAULT_TABLE table=default.Table_5468f28974964427aa77e0d4809ca02c [id=b71cd57d9d9c40c79def420949196b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.581518 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c70f54de70b4d0e94d25239b097f8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.584733 1159 tablet_bootstrap.cc:492] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:48.584873 1159 ts_tablet_manager.cc:1397] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.151s user 0.005s sys 0.005s
I20250629 01:51:48.585503 1159 raft_consensus.cc:357] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.585672 1159 raft_consensus.cc:383] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.585741 1159 raft_consensus.cc:738] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.585964 1159 consensus_queue.cc:260] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.590477 1159 ts_tablet_manager.cc:1428] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:48.590970 1155 tablet_bootstrap.cc:492] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.591163 1155 ts_tablet_manager.cc:1397] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250629 01:51:48.591761 886 tablet_service.cc:1468] Processing CreateTablet for tablet 3c70f54de70b4d0e94d25239b097f8fa (DEFAULT_TABLE table=default.Table_5468f28974964427aa77e0d4809ca02c [id=b71cd57d9d9c40c79def420949196b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.592123 1158 ts_tablet_manager.cc:1918] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.592649 1155 raft_consensus.cc:357] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.592849 1155 raft_consensus.cc:383] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.592952 1155 raft_consensus.cc:738] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.593257 1155 consensus_queue.cc:260] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.594070 743 tablet_service.cc:1468] Processing CreateTablet for tablet 13cbdc69cebd4117ab5d35299d734797 (DEFAULT_TABLE table=default.Table_f8e69accdc0a4e35a1d60c963e638c42 [id=5aa6b128f37a4cd699ccf48b509e8955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.594414 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13cbdc69cebd4117ab5d35299d734797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.594848 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2175d9af5064903ab1c97c2035e2527" } modify_external_catalogs: true
I20250629 01:51:48.595052 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2175d9af5064903ab1c97c2035e2527" } modify_external_catalogs: true
I20250629 01:51:48.595137 1155 ts_tablet_manager.cc:1428] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:48.595407 1155 tablet_bootstrap.cc:492] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.596858 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:48.593298 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 3c70f54de70b4d0e94d25239b097f8fa (DEFAULT_TABLE table=default.Table_5468f28974964427aa77e0d4809ca02c [id=b71cd57d9d9c40c79def420949196b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.598086 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c70f54de70b4d0e94d25239b097f8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.599304 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5e27d74509cc4dc9a4f0566c61348aa3" } modify_external_catalogs: true
I20250629 01:51:48.599450 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5e27d74509cc4dc9a4f0566c61348aa3" } modify_external_catalogs: true
I20250629 01:51:48.602135 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 13cbdc69cebd4117ab5d35299d734797 (DEFAULT_TABLE table=default.Table_f8e69accdc0a4e35a1d60c963e638c42 [id=5aa6b128f37a4cd699ccf48b509e8955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.602416 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13cbdc69cebd4117ab5d35299d734797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.603453 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c70f54de70b4d0e94d25239b097f8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.605808 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 897a4cefefe04f06b47070ec7fa1a33b
I20250629 01:51:48.607007 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.607264 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.609043 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4d9ef77181ba48f3a6de16b945c8df1a
I20250629 01:51:48.609763 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.610355 1154 tablet_bootstrap.cc:492] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.610517 1154 ts_tablet_manager.cc:1397] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.006s
I20250629 01:51:48.611060 1154 raft_consensus.cc:357] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.611245 1154 raft_consensus.cc:383] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.611236 880 tablet_service.cc:1468] Processing CreateTablet for tablet 13cbdc69cebd4117ab5d35299d734797 (DEFAULT_TABLE table=default.Table_f8e69accdc0a4e35a1d60c963e638c42 [id=5aa6b128f37a4cd699ccf48b509e8955]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.611335 1154 raft_consensus.cc:738] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.611639 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13cbdc69cebd4117ab5d35299d734797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.611657 1154 consensus_queue.cc:260] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.612059 1154 ts_tablet_manager.cc:1428] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:48.612197 1158 log.cc:1199] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/96324ddf32a1460b8fc37e2c324649e6
I20250629 01:51:48.612584 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.612484 1154 tablet_bootstrap.cc:492] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.612766 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.614208 1154 tablet_bootstrap.cc:654] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.614464 1158 ts_tablet_manager.cc:1939] T 96324ddf32a1460b8fc37e2c324649e6 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.616118 1158 tablet_replica.cc:331] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.616324 1158 raft_consensus.cc:2241] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.616516 1158 raft_consensus.cc:2270] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.616966 1158 ts_tablet_manager.cc:1905] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:48.619102 556 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:56714 (request call id 22) took 912 ms (client timeout 1000 ms). Trace:
W20250629 01:51:48.619240 556 rpcz_store.cc:269] 0629 01:51:47.707011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.707062 (+ 51us) service_pool.cc:225] Handling call
0629 01:51:48.619045 (+911983us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250629 01:51:48.621527 524 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:51799 (request call id 170) took 863 ms (client timeout 1000 ms). Trace:
W20250629 01:51:48.621630 524 rpcz_store.cc:269] 0629 01:51:47.758516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.760540 (+ 2024us) service_pool.cc:225] Handling call
0629 01:51:48.621468 (+860928us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250629 01:51:48.622910 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu with pid 494
I20250629 01:51:48.624737 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_91cf07f495b64649afe5b6d70efa7dba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.625192 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91cf07f495b64649afe5b6d70efa7dba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.626683 1158 ts_tablet_manager.cc:1918] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.626789 1158 log.cc:1199] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:48.627141 1158 ts_tablet_manager.cc:1939] T 69980d5bf37d4001a6196b22c34ccc94 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.627817 1158 tablet_replica.cc:331] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.627995 1158 raft_consensus.cc:2241] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.628150 1158 raft_consensus.cc:2270] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.628575 1158 ts_tablet_manager.cc:1905] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.628556 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 69980d5bf37d4001a6196b22c34ccc94 (table default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 [id=fd2351d583834130bb8213424bf96abb]) successfully deleted
I20250629 01:51:48.629091 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_28e044c713374ff284d41c0262e1bbc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.629221 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_82d6223100ea4aba8941923226f2baa6" } modify_external_catalogs: true
I20250629 01:51:48.629383 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_82d6223100ea4aba8941923226f2baa6" } modify_external_catalogs: true
W20250629 01:51:48.629556 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28e044c713374ff284d41c0262e1bbc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.629609 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 789314589f284283a0d37831f5651d3f (DEFAULT_TABLE table=default.Table_5fba3653ae2c4de78346d2983120e610 [id=9cf2c6167fc244c39ef27c4fc17cd42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.629913 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789314589f284283a0d37831f5651d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.631290 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a8bede9e061941eaabacba34109b16ab" } modify_external_catalogs: true
I20250629 01:51:48.631415 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a8bede9e061941eaabacba34109b16ab" } modify_external_catalogs: true
I20250629 01:51:48.632376 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 1336aea8f6b1436a971766d92d3c1197 (DEFAULT_TABLE table=default.Table_8aceaa540cf94e83ba9250c612cab7a5 [id=a516e76dca9b418f90d5de6e5741cef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.632772 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1336aea8f6b1436a971766d92d3c1197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.637176 889 tablet_service.cc:1468] Processing CreateTablet for tablet 250ce9c0b27c40d996db138d0acb28d7 (DEFAULT_TABLE table=default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a [id=a9def3c27a1f4613a18e103cc015128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.637238 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 250ce9c0b27c40d996db138d0acb28d7 (DEFAULT_TABLE table=default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a [id=a9def3c27a1f4613a18e103cc015128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.637516 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250ce9c0b27c40d996db138d0acb28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.638319 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250ce9c0b27c40d996db138d0acb28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.638522 1022 tablet_service.cc:1468] Processing CreateTablet for tablet cfb025535ecd4cf88681c4951a56ba29 (DEFAULT_TABLE table=default.Table_caef1729000a42b482b4da2dcd58f39a [id=64f1e007d5674b17be6e2575c58ccd8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.638814 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb025535ecd4cf88681c4951a56ba29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.640347 894 tablet_service.cc:1468] Processing CreateTablet for tablet cfb025535ecd4cf88681c4951a56ba29 (DEFAULT_TABLE table=default.Table_caef1729000a42b482b4da2dcd58f39a [id=64f1e007d5674b17be6e2575c58ccd8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.640636 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb025535ecd4cf88681c4951a56ba29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.644346 888 tablet_service.cc:1468] Processing CreateTablet for tablet 789314589f284283a0d37831f5651d3f (DEFAULT_TABLE table=default.Table_5fba3653ae2c4de78346d2983120e610 [id=9cf2c6167fc244c39ef27c4fc17cd42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.644680 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789314589f284283a0d37831f5651d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.647542 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b75d33594c994363bdb2558eb0c6ca3d
I20250629 01:51:48.648211 885 tablet_service.cc:1468] Processing CreateTablet for tablet 62369ffc1d4a4628bb24254877d6dee6 (DEFAULT_TABLE table=default.Table_14ca2bdaa17c48b59043ada398aa4bc4 [id=5c192ab5cd2a4456b32cab788c8d015c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.648497 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62369ffc1d4a4628bb24254877d6dee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.648578 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1336aea8f6b1436a971766d92d3c1197 (DEFAULT_TABLE table=default.Table_8aceaa540cf94e83ba9250c612cab7a5 [id=a516e76dca9b418f90d5de6e5741cef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.648799 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 258994dd24804f4aaa5e299436fac9b0
I20250629 01:51:48.648870 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1336aea8f6b1436a971766d92d3c1197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.650295 1710 raft_consensus.cc:491] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.650413 1710 raft_consensus.cc:513] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.652582 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet b75d33594c994363bdb2558eb0c6ca3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.652779 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 258994dd24804f4aaa5e299436fac9b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.653834 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2c0e663e50e24133b9794fc3ab9022c6" } new_table_name: "default.Table_9e68c12ac74247ffb80d0ee3b879d226" modify_external_catalogs: true
I20250629 01:51:48.656431 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 62369ffc1d4a4628bb24254877d6dee6 (DEFAULT_TABLE table=default.Table_14ca2bdaa17c48b59043ada398aa4bc4 [id=5c192ab5cd2a4456b32cab788c8d015c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.656689 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62369ffc1d4a4628bb24254877d6dee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.657334 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_be093346e01e4a688e5ede68f9e96fbd" } new_table_name: "default.Table_75440a0f96a3433f98a5840644b52d1f" modify_external_catalogs: true
I20250629 01:51:48.660744 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c17d83489bb0493ab307c56ab6212e37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.661324 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c17d83489bb0493ab307c56ab6212e37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.664156 1710 leader_election.cc:290] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.664872 1029 tablet_service.cc:1468] Processing CreateTablet for tablet fb48f4a2d9724300b9c00df6e12e92b0 (DEFAULT_TABLE table=default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.665153 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb48f4a2d9724300b9c00df6e12e92b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.665701 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 4cc7e839241e447db039b261421a7fef (DEFAULT_TABLE table=default.Table_c5c42b0786ae4ba8ae6d0165aae1bc60 [id=226b83f54c544b0fb4e3181dc77875d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.665958 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc7e839241e447db039b261421a7fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.666754 1158 ts_tablet_manager.cc:1918] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.666886 1158 log.cc:1199] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/dd709b557b334434a6c2c4a60a996342
I20250629 01:51:48.667193 1158 ts_tablet_manager.cc:1939] T dd709b557b334434a6c2c4a60a996342 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.668043 1158 tablet_replica.cc:331] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.668031 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet dd709b557b334434a6c2c4a60a996342 (table default.Table_b5f98a84fe7341ac8e6b11b04066323c [id=dd8b9804d0144a879c539614acde7240]) successfully deleted
I20250629 01:51:48.668229 1158 raft_consensus.cc:2241] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.668354 1158 raft_consensus.cc:2270] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.668742 1158 ts_tablet_manager.cc:1905] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.669994 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b822effa18c64fe28fcfc3ff8db8c6f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.670405 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b822effa18c64fe28fcfc3ff8db8c6f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:48.671865 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c6239dd09a343d59da51d114f48658a consensus state has no leader
I20250629 01:51:48.672489 887 tablet_service.cc:1515] Processing DeleteTablet for tablet b75d33594c994363bdb2558eb0c6ca3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.672693 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 258994dd24804f4aaa5e299436fac9b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.672974 883 tablet_service.cc:1468] Processing CreateTablet for tablet 4cc7e839241e447db039b261421a7fef (DEFAULT_TABLE table=default.Table_c5c42b0786ae4ba8ae6d0165aae1bc60 [id=226b83f54c544b0fb4e3181dc77875d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.673442 1158 ts_tablet_manager.cc:1918] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.673523 1158 log.cc:1199] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0c712d978a3848b9bb3abdbce8314dec
I20250629 01:51:48.673532 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc7e839241e447db039b261421a7fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.673753 1158 ts_tablet_manager.cc:1939] T 0c712d978a3848b9bb3abdbce8314dec P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.674054 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.672549 884 tablet_service.cc:1468] Processing CreateTablet for tablet fb48f4a2d9724300b9c00df6e12e92b0 (DEFAULT_TABLE table=default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:48.674428 988 leader_election.cc:343] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:48.675771 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76118a6e684448b09f244d44c909ccaa consensus state has no leader
I20250629 01:51:48.675740 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb48f4a2d9724300b9c00df6e12e92b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.676798 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c9445c5e7b1a4c62af297a2184934b9a" } modify_external_catalogs: true
I20250629 01:51:48.676959 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c9445c5e7b1a4c62af297a2184934b9a" } modify_external_catalogs: true
I20250629 01:51:48.678434 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d5c8e3cdfbb743438fd806f1e2dd5bf7 (DEFAULT_TABLE table=default.Table_b8599c95728d43dd898fbdfcb4fe5db3 [id=3f630648e78d44fabe953cbacd9a9602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.678504 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0c712d978a3848b9bb3abdbce8314dec (table default.Table_ca2bcac172414e6c9f264cb2202a76fa [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:48.678946 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c8e3cdfbb743438fd806f1e2dd5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.679225 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 021da4395e414413b9b11ebcfbffcd19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 021da4395e414413b9b11ebcfbffcd19 already in progress: creating tablet
I20250629 01:51:48.679723 1158 tablet_replica.cc:331] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.680178 1158 raft_consensus.cc:2241] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.680225 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.680474 1158 raft_consensus.cc:2270] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.681087 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 17acb9bcd7aa4a508d1695474a1b56df (DEFAULT_TABLE table=default.Table_541d81feb03a418780ca4f7da1ed1d4c [id=024fed167fdd4acda0d6f1224e32b18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.681339 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17acb9bcd7aa4a508d1695474a1b56df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.682624 745 tablet_service.cc:1468] Processing CreateTablet for tablet 789314589f284283a0d37831f5651d3f (DEFAULT_TABLE table=default.Table_5fba3653ae2c4de78346d2983120e610 [id=9cf2c6167fc244c39ef27c4fc17cd42e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.682952 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 789314589f284283a0d37831f5651d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.680887 1158 ts_tablet_manager.cc:1905] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:48.685271 986 leader_election.cc:343] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.685369 986 leader_election.cc:304] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.686093 741 tablet_service.cc:1468] Processing CreateTablet for tablet d5c8e3cdfbb743438fd806f1e2dd5bf7 (DEFAULT_TABLE table=default.Table_b8599c95728d43dd898fbdfcb4fe5db3 [id=3f630648e78d44fabe953cbacd9a9602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.686260 1710 raft_consensus.cc:2747] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.686195 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4de24a8e2d8f4c888ab0adb848c6deed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.686553 755 tablet_service.cc:1468] Processing CreateTablet for tablet cfb025535ecd4cf88681c4951a56ba29 (DEFAULT_TABLE table=default.Table_caef1729000a42b482b4da2dcd58f39a [id=64f1e007d5674b17be6e2575c58ccd8d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:48.686611 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4de24a8e2d8f4c888ab0adb848c6deed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.686858 744 tablet_service.cc:1468] Processing CreateTablet for tablet 250ce9c0b27c40d996db138d0acb28d7 (DEFAULT_TABLE table=default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a [id=a9def3c27a1f4613a18e103cc015128f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.687150 887 tablet_service.cc:1468] Processing CreateTablet for tablet 17acb9bcd7aa4a508d1695474a1b56df (DEFAULT_TABLE table=default.Table_541d81feb03a418780ca4f7da1ed1d4c [id=024fed167fdd4acda0d6f1224e32b18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.687458 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17acb9bcd7aa4a508d1695474a1b56df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.688409 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:48.688432 740 tablet_service.cc:1468] Processing CreateTablet for tablet fb48f4a2d9724300b9c00df6e12e92b0 (DEFAULT_TABLE table=default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.688663 752 tablet_service.cc:1468] Processing CreateTablet for tablet 4cc7e839241e447db039b261421a7fef (DEFAULT_TABLE table=default.Table_c5c42b0786ae4ba8ae6d0165aae1bc60 [id=226b83f54c544b0fb4e3181dc77875d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.689509 738 tablet_service.cc:1515] Processing DeleteTablet for tablet b75d33594c994363bdb2558eb0c6ca3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.690071 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 258994dd24804f4aaa5e299436fac9b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.690409 751 tablet_service.cc:1468] Processing CreateTablet for tablet 17acb9bcd7aa4a508d1695474a1b56df (DEFAULT_TABLE table=default.Table_541d81feb03a418780ca4f7da1ed1d4c [id=024fed167fdd4acda0d6f1224e32b18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.690629 891 tablet_service.cc:1468] Processing CreateTablet for tablet d5c8e3cdfbb743438fd806f1e2dd5bf7 (DEFAULT_TABLE table=default.Table_b8599c95728d43dd898fbdfcb4fe5db3 [id=3f630648e78d44fabe953cbacd9a9602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.690663 742 tablet_service.cc:1468] Processing CreateTablet for tablet 62369ffc1d4a4628bb24254877d6dee6 (DEFAULT_TABLE table=default.Table_14ca2bdaa17c48b59043ada398aa4bc4 [id=5c192ab5cd2a4456b32cab788c8d015c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.690899 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c8e3cdfbb743438fd806f1e2dd5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.691877 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5c8e3cdfbb743438fd806f1e2dd5bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.692821 749 tablet_service.cc:1468] Processing CreateTablet for tablet 1336aea8f6b1436a971766d92d3c1197 (DEFAULT_TABLE table=default.Table_8aceaa540cf94e83ba9250c612cab7a5 [id=a516e76dca9b418f90d5de6e5741cef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.693408 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.693732 1710 raft_consensus.cc:491] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.693807 1156 ts_tablet_manager.cc:1918] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.693830 1710 raft_consensus.cc:513] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:48.693907 1156 log.cc:1199] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0b7a640451934bff8672cd88dc86c529
I20250629 01:51:48.694293 1710 leader_election.cc:290] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:48.694417 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb48f4a2d9724300b9c00df6e12e92b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.697510 881 tablet_service.cc:1468] Processing CreateTablet for tablet 9c1dc6d6d2b1458cb57875d2e25354bc (DEFAULT_TABLE table=default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.697381 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 9c1dc6d6d2b1458cb57875d2e25354bc (DEFAULT_TABLE table=default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.697786 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c1dc6d6d2b1458cb57875d2e25354bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.698185 739 tablet_service.cc:1468] Processing CreateTablet for tablet 9c1dc6d6d2b1458cb57875d2e25354bc (DEFAULT_TABLE table=default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.701969 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_412fa53923654534afbfbdb488806ed2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.702421 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_412fa53923654534afbfbdb488806ed2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.702606 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d387af53209147a6bac7afeda9a411ab (DEFAULT_TABLE table=default.Table_c1fb2548e78248e98b54bfb7f01c0e51 [id=a2b08c6d970a45709f099af55d84f522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.702860 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d387af53209147a6bac7afeda9a411ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.702858 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1336aea8f6b1436a971766d92d3c1197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.703577 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.703765 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c1dc6d6d2b1458cb57875d2e25354bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.704119 986 leader_election.cc:343] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.704334 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c1dc6d6d2b1458cb57875d2e25354bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.704636 893 tablet_service.cc:1468] Processing CreateTablet for tablet d387af53209147a6bac7afeda9a411ab (DEFAULT_TABLE table=default.Table_c1fb2548e78248e98b54bfb7f01c0e51 [id=a2b08c6d970a45709f099af55d84f522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.705081 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0685872de4674119b2386ac286878d35" } new_table_name: "default.Table_2657fd4cf2044332bcbf4e1027d13bee" modify_external_catalogs: true
I20250629 01:51:48.705338 738 tablet_service.cc:1468] Processing CreateTablet for tablet 98031e73f9ae4d1e9aa2f75ee0bb7e18 (DEFAULT_TABLE table=default.Table_0cf9b58194d14b1a8f94c2c12e54a36b [id=b65fcdb8452c454698377ea184887ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.705621 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98031e73f9ae4d1e9aa2f75ee0bb7e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.705984 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7b6e3def9c71407cbd04500227e036ad" } new_table_name: "default.Table_1f284b2051314392a75e40bc77ef6a3f" modify_external_catalogs: true
I20250629 01:51:48.706123 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d387af53209147a6bac7afeda9a411ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.706844 1664 raft_consensus.cc:491] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.703510 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.706962 1664 raft_consensus.cc:513] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:48.707260 988 leader_election.cc:343] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.707343 988 leader_election.cc:304] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.707504 1664 leader_election.cc:290] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.707656 1710 raft_consensus.cc:2747] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.707890 747 tablet_service.cc:1468] Processing CreateTablet for tablet d387af53209147a6bac7afeda9a411ab (DEFAULT_TABLE table=default.Table_c1fb2548e78248e98b54bfb7f01c0e51 [id=a2b08c6d970a45709f099af55d84f522]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.708200 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d387af53209147a6bac7afeda9a411ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.711412 877 tablet_service.cc:1468] Processing CreateTablet for tablet 98031e73f9ae4d1e9aa2f75ee0bb7e18 (DEFAULT_TABLE table=default.Table_0cf9b58194d14b1a8f94c2c12e54a36b [id=b65fcdb8452c454698377ea184887ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.711695 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98031e73f9ae4d1e9aa2f75ee0bb7e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.711830 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 98031e73f9ae4d1e9aa2f75ee0bb7e18 (DEFAULT_TABLE table=default.Table_0cf9b58194d14b1a8f94c2c12e54a36b [id=b65fcdb8452c454698377ea184887ac2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.712082 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98031e73f9ae4d1e9aa2f75ee0bb7e18. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.712659 712 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: fca257d6cb614110948e1045dee44bdc
I20250629 01:51:48.712852 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250ce9c0b27c40d996db138d0acb28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.716197 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb025535ecd4cf88681c4951a56ba29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.716293 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_974936d8b1084dcc8e5befb81e238b37" } new_table_name: "default.Table_d7032fa31e084ecc8861910661af6259" modify_external_catalogs: true
I20250629 01:51:48.719141 746 tablet_service.cc:1468] Processing CreateTablet for tablet 0f05d54fc9e649faabee85a418ebf2bb (DEFAULT_TABLE table=default.Table_415b32694788453db597034d0865a18a [id=e0e91a7341a44357b8c4b44c08b30c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.719434 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f05d54fc9e649faabee85a418ebf2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.720644 887 tablet_service.cc:1468] Processing CreateTablet for tablet 0f05d54fc9e649faabee85a418ebf2bb (DEFAULT_TABLE table=default.Table_415b32694788453db597034d0865a18a [id=e0e91a7341a44357b8c4b44c08b30c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.720953 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f05d54fc9e649faabee85a418ebf2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.722795 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cc7e839241e447db039b261421a7fef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.724447 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_709271d3d173470ab9aace403ba9d5e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.724871 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_709271d3d173470ab9aace403ba9d5e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.724835 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 0f05d54fc9e649faabee85a418ebf2bb (DEFAULT_TABLE table=default.Table_415b32694788453db597034d0865a18a [id=e0e91a7341a44357b8c4b44c08b30c43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.725085 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f05d54fc9e649faabee85a418ebf2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.725392 754 tablet_service.cc:1468] Processing CreateTablet for tablet 84b0092d5dbb4f1a93eebd980959362b (DEFAULT_TABLE table=default.Table_705f111850114af4b8ada44b31486222 [id=b1c1a70a5d614e4aa85bc31e95077fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.725697 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b0092d5dbb4f1a93eebd980959362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.725955 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17acb9bcd7aa4a508d1695474a1b56df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.731492 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62369ffc1d4a4628bb24254877d6dee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.732271 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2474fd6ed64c40278b389fe3aabc1765"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.732702 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2474fd6ed64c40278b389fe3aabc1765 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.735496 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 84b0092d5dbb4f1a93eebd980959362b (DEFAULT_TABLE table=default.Table_705f111850114af4b8ada44b31486222 [id=b1c1a70a5d614e4aa85bc31e95077fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.735744 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b0092d5dbb4f1a93eebd980959362b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.740224 710 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: fca257d6cb614110948e1045dee44bdc
I20250629 01:51:48.740326 710 leader_election.cc:304] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:48.740679 1664 raft_consensus.cc:2747] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:48.742360 743 tablet_service.cc:1468] Processing CreateTablet for tablet 871086e50583453cadcc60ccd8afcba6 (DEFAULT_TABLE table=default.Table_c9445c5e7b1a4c62af297a2184934b9a [id=337ba361eef849ceb4e7a24084f487f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.742642 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871086e50583453cadcc60ccd8afcba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.742825 890 tablet_service.cc:1468] Processing CreateTablet for tablet 871086e50583453cadcc60ccd8afcba6 (DEFAULT_TABLE table=default.Table_c9445c5e7b1a4c62af297a2184934b9a [id=337ba361eef849ceb4e7a24084f487f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.743144 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871086e50583453cadcc60ccd8afcba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.745533 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_bbfbc9cc8c0f441c8718c1ea68f3a287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.745918 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbfbc9cc8c0f441c8718c1ea68f3a287 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.747979 878 tablet_service.cc:1468] Processing CreateTablet for tablet d87cc2503f2e4aaa861c42ad459ddffa (DEFAULT_TABLE table=default.Table_dcfccdde4f3941dd8db4b2db608ed1d9 [id=359cf8466c994daab46d5d241eeb4851]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.748332 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87cc2503f2e4aaa861c42ad459ddffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.750258 1158 ts_tablet_manager.cc:1918] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.750348 1158 log.cc:1199] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/28eaa55b3b564e2e9cf698fda98a4c8a
I20250629 01:51:48.750608 1158 ts_tablet_manager.cc:1939] T 28eaa55b3b564e2e9cf698fda98a4c8a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:48.752009 1710 raft_consensus.cc:491] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.752135 882 tablet_service.cc:1468] Processing CreateTablet for tablet 8a5c7722575e452aa009931d63637b93 (DEFAULT_TABLE table=default.Table_f9574ad3b53b407d9a3352687623467a [id=5d76408807934aaa9a36f35df7918580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.752146 1710 raft_consensus.cc:513] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.752418 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5c7722575e452aa009931d63637b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.752633 1710 leader_election.cc:290] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:48.753228 877 tablet_service.cc:1468] Processing CreateTablet for tablet 84b0092d5dbb4f1a93eebd980959362b (DEFAULT_TABLE table=default.Table_705f111850114af4b8ada44b31486222 [id=b1c1a70a5d614e4aa85bc31e95077fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.753520 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84b0092d5dbb4f1a93eebd980959362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.755426 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:48.755764 988 leader_election.cc:343] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.755782 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d87cc2503f2e4aaa861c42ad459ddffa (DEFAULT_TABLE table=default.Table_dcfccdde4f3941dd8db4b2db608ed1d9 [id=359cf8466c994daab46d5d241eeb4851]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.756233 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87cc2503f2e4aaa861c42ad459ddffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.757481 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 871086e50583453cadcc60ccd8afcba6
I20250629 01:51:48.758319 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 871086e50583453cadcc60ccd8afcba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.758347 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 871086e50583453cadcc60ccd8afcba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.760275 1155 tablet_bootstrap.cc:654] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.760380 1159 tablet_bootstrap.cc:492] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:48.760331 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 871086e50583453cadcc60ccd8afcba6 (DEFAULT_TABLE table=default.Table_c9445c5e7b1a4c62af297a2184934b9a [id=337ba361eef849ceb4e7a24084f487f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.760594 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 871086e50583453cadcc60ccd8afcba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.760962 1156 ts_tablet_manager.cc:1939] T 0b7a640451934bff8672cd88dc86c529 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.761349 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 8a5c7722575e452aa009931d63637b93 (DEFAULT_TABLE table=default.Table_f9574ad3b53b407d9a3352687623467a [id=5d76408807934aaa9a36f35df7918580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.761611 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5c7722575e452aa009931d63637b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.761924 1159 tablet_bootstrap.cc:654] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.763119 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.764612 1156 tablet_replica.cc:331] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.764808 742 tablet_service.cc:1468] Processing CreateTablet for tablet d87cc2503f2e4aaa861c42ad459ddffa (DEFAULT_TABLE table=default.Table_dcfccdde4f3941dd8db4b2db608ed1d9 [id=359cf8466c994daab46d5d241eeb4851]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:48.765017 986 leader_election.cc:343] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.765120 986 leader_election.cc:304] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.766243 1156 raft_consensus.cc:2241] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.766388 1156 raft_consensus.cc:2270] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.767125 1156 ts_tablet_manager.cc:1905] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.771906 896 tablet_service.cc:1468] Processing CreateTablet for tablet 621abed696cd4284bf8ee0223df312ee (DEFAULT_TABLE table=default.Table_5da0f7f4f413447e84b57be34c67019b [id=03e506095412488fa889867d23a0303c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.771997 1710 raft_consensus.cc:2747] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.772264 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 621abed696cd4284bf8ee0223df312ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.772403 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:48.775653 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 621abed696cd4284bf8ee0223df312ee (DEFAULT_TABLE table=default.Table_5da0f7f4f413447e84b57be34c67019b [id=03e506095412488fa889867d23a0303c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.775966 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 621abed696cd4284bf8ee0223df312ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.776444 895 tablet_service.cc:1468] Processing CreateTablet for tablet a0769f77ea234dd7bccc2244c08b512b (DEFAULT_TABLE table=default.Table_edae10908319498e8780ce62b5f260c7 [id=28289197683b442fa652de9286bc4a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.776741 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0769f77ea234dd7bccc2244c08b512b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.776928 750 tablet_service.cc:1468] Processing CreateTablet for tablet 621abed696cd4284bf8ee0223df312ee (DEFAULT_TABLE table=default.Table_5da0f7f4f413447e84b57be34c67019b [id=03e506095412488fa889867d23a0303c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.777274 757 tablet_service.cc:1468] Processing CreateTablet for tablet a0769f77ea234dd7bccc2244c08b512b (DEFAULT_TABLE table=default.Table_edae10908319498e8780ce62b5f260c7 [id=28289197683b442fa652de9286bc4a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.779307 751 tablet_service.cc:1468] Processing CreateTablet for tablet 8a5c7722575e452aa009931d63637b93 (DEFAULT_TABLE table=default.Table_f9574ad3b53b407d9a3352687623467a [id=5d76408807934aaa9a36f35df7918580]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.780426 879 tablet_service.cc:1468] Processing CreateTablet for tablet 39d2adf801934179825f96a5f7b3c303 (DEFAULT_TABLE table=default.Table_951aa64d945448d4b3c17187cf5bff48 [id=12bb1ae995cc4e2a884a4d3c43f814b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.780735 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39d2adf801934179825f96a5f7b3c303. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.791802 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d87cc2503f2e4aaa861c42ad459ddffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.795265 1156 ts_tablet_manager.cc:1918] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.795516 1156 log.cc:1199] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4dbe70d4e4e74c6ba7db2766bd78c8ad
I20250629 01:51:48.795868 1156 ts_tablet_manager.cc:1939] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.796308 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 621abed696cd4284bf8ee0223df312ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.799932 451 master_service.cc:948] ReplaceTablet: received request to replace tablet bfa48f94ac0e48e39578a6e8af5c8e9c from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.800403 1156 tablet_replica.cc:331] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.800499 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0769f77ea234dd7bccc2244c08b512b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.800591 1156 raft_consensus.cc:2241] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.800741 1156 raft_consensus.cc:2270] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.801402 1156 ts_tablet_manager.cc:1905] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.804934 448 master_service.cc:948] ReplaceTablet: received request to replace tablet ea00b4559e03410885180286b5e9fd10 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.808365 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a5c7722575e452aa009931d63637b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.812647 1156 ts_tablet_manager.cc:1918] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.812765 1156 log.cc:1199] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/96324ddf32a1460b8fc37e2c324649e6
I20250629 01:51:48.813103 1156 ts_tablet_manager.cc:1939] T 96324ddf32a1460b8fc37e2c324649e6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.813306 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0b7a640451934bff8672cd88dc86c529 (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]) successfully deleted
W20250629 01:51:48.813563 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0827ec622ff04ca6b2fec50e40b9e1de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0827ec622ff04ca6b2fec50e40b9e1de already in progress: creating tablet
W20250629 01:51:48.813751 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3070ce8c467a47b0a833374450911291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3070ce8c467a47b0a833374450911291 already in progress: creating tablet
W20250629 01:51:48.813956 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 28eaa55b3b564e2e9cf698fda98a4c8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28eaa55b3b564e2e9cf698fda98a4c8a already in progress: creating tablet
I20250629 01:51:48.814222 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad (table default.Table_adc0a47bc8504a73aa8b9ede02097dd9 [id=604220139d4f4b31bfaca78aa2b32f29]) successfully deleted
I20250629 01:51:48.816195 747 tablet_service.cc:1468] Processing CreateTablet for tablet 39d2adf801934179825f96a5f7b3c303 (DEFAULT_TABLE table=default.Table_951aa64d945448d4b3c17187cf5bff48 [id=12bb1ae995cc4e2a884a4d3c43f814b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.816521 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39d2adf801934179825f96a5f7b3c303. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.816737 741 tablet_service.cc:1468] Processing CreateTablet for tablet 421ffa84a7664826a6cbc303653fd00b (DEFAULT_TABLE table=default.Table_c2146a78ba3746aea6dc9d3614b1e5c7 [id=fe8ee40f067b49d296eeb859a1169306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.816749 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 53660707e9204e2989a63b46d007140a (DEFAULT_TABLE table=default.Table_1c68a0f197b241f781976418aed57d90 [id=126469234ced48d3b0cc236da00e2768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.817034 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421ffa84a7664826a6cbc303653fd00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.817045 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53660707e9204e2989a63b46d007140a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.817761 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 50ab66cf5f2048fcb80e323c0935d83f (DEFAULT_TABLE table=default.Table_69f7afd6dbc24708810b9a825012ae62 [id=ed564ef3e87a498aa51ebcac75358003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.818024 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ab66cf5f2048fcb80e323c0935d83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.820379 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 421ffa84a7664826a6cbc303653fd00b (DEFAULT_TABLE table=default.Table_c2146a78ba3746aea6dc9d3614b1e5c7 [id=fe8ee40f067b49d296eeb859a1169306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.820644 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421ffa84a7664826a6cbc303653fd00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.820691 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 39d2adf801934179825f96a5f7b3c303 (DEFAULT_TABLE table=default.Table_951aa64d945448d4b3c17187cf5bff48 [id=12bb1ae995cc4e2a884a4d3c43f814b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.821002 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3699a1e4f7ec403ba943711d0c3e6588"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.821419 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3699a1e4f7ec403ba943711d0c3e6588 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.822182 887 tablet_service.cc:1468] Processing CreateTablet for tablet 53660707e9204e2989a63b46d007140a (DEFAULT_TABLE table=default.Table_1c68a0f197b241f781976418aed57d90 [id=126469234ced48d3b0cc236da00e2768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.822474 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53660707e9204e2989a63b46d007140a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.823745 881 tablet_service.cc:1468] Processing CreateTablet for tablet 50ab66cf5f2048fcb80e323c0935d83f (DEFAULT_TABLE table=default.Table_69f7afd6dbc24708810b9a825012ae62 [id=ed564ef3e87a498aa51ebcac75358003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.824255 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ab66cf5f2048fcb80e323c0935d83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.825016 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39d2adf801934179825f96a5f7b3c303. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.825193 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a0769f77ea234dd7bccc2244c08b512b (DEFAULT_TABLE table=default.Table_edae10908319498e8780ce62b5f260c7 [id=28289197683b442fa652de9286bc4a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.826007 885 tablet_service.cc:1468] Processing CreateTablet for tablet 421ffa84a7664826a6cbc303653fd00b (DEFAULT_TABLE table=default.Table_c2146a78ba3746aea6dc9d3614b1e5c7 [id=fe8ee40f067b49d296eeb859a1169306]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.826030 1710 consensus_queue.cc:1035] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:48.826440 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421ffa84a7664826a6cbc303653fd00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.826668 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 96324ddf32a1460b8fc37e2c324649e6 (table default.Table_8c1fbbaabe004466858cce35b18194b1 [id=700ef08e094b44369154076e65d9ea94]) successfully deleted
W20250629 01:51:48.826879 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 897a4cefefe04f06b47070ec7fa1a33b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897a4cefefe04f06b47070ec7fa1a33b already in progress: creating tablet
I20250629 01:51:48.826907 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0769f77ea234dd7bccc2244c08b512b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:48.827121 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b75d33594c994363bdb2558eb0c6ca3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b75d33594c994363bdb2558eb0c6ca3d already in progress: creating tablet
W20250629 01:51:48.827303 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 258994dd24804f4aaa5e299436fac9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 258994dd24804f4aaa5e299436fac9b0 already in progress: creating tablet
W20250629 01:51:48.827493 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4d9ef77181ba48f3a6de16b945c8df1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d9ef77181ba48f3a6de16b945c8df1a already in progress: creating tablet
W20250629 01:51:48.830788 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0467b153057f491e81681fa399a2a1a9 consensus state has no leader
I20250629 01:51:48.831153 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5d35d0525f924ed692497dbaf2287e8e" } modify_external_catalogs: true
I20250629 01:51:48.831305 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5d35d0525f924ed692497dbaf2287e8e" } modify_external_catalogs: true
I20250629 01:51:48.832633 749 tablet_service.cc:1468] Processing CreateTablet for tablet 53660707e9204e2989a63b46d007140a (DEFAULT_TABLE table=default.Table_1c68a0f197b241f781976418aed57d90 [id=126469234ced48d3b0cc236da00e2768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.832937 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53660707e9204e2989a63b46d007140a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.837975 1159 tablet_bootstrap.cc:492] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:48.838032 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 871086e50583453cadcc60ccd8afcba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.838168 1159 ts_tablet_manager.cc:1397] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.078s user 0.001s sys 0.002s
I20250629 01:51:48.838706 743 tablet_service.cc:1468] Processing CreateTablet for tablet 50ab66cf5f2048fcb80e323c0935d83f (DEFAULT_TABLE table=default.Table_69f7afd6dbc24708810b9a825012ae62 [id=ed564ef3e87a498aa51ebcac75358003]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:48.838992 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ab66cf5f2048fcb80e323c0935d83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:48.838907 1159 raft_consensus.cc:357] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.839203 1159 raft_consensus.cc:383] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.839308 1159 raft_consensus.cc:738] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.839529 1159 consensus_queue.cc:260] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.840785 1159 ts_tablet_manager.cc:1428] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:51:48.842502 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1961bae4492d41e8b6c8536a9f324977" } modify_external_catalogs: true
I20250629 01:51:48.842677 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1961bae4492d41e8b6c8536a9f324977" } modify_external_catalogs: true
I20250629 01:51:48.843353 1159 tablet_bootstrap.cc:492] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:48.854204 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 871086e50583453cadcc60ccd8afcba6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871086e50583453cadcc60ccd8afcba6 already in progress: creating tablet
I20250629 01:51:48.858649 1157 ts_tablet_manager.cc:1918] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:48.858748 1157 log.cc:1199] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0f40aefcb7e14612bf2d435dfa37d66e
I20250629 01:51:48.859089 1157 ts_tablet_manager.cc:1939] T 0f40aefcb7e14612bf2d435dfa37d66e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:48.863265 1159 tablet_bootstrap.cc:654] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.863260 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1c1f88cfa7a84329b2ffe152996d178e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.863713 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c1f88cfa7a84329b2ffe152996d178e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.864317 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bfa48f94ac0e48e39578a6e8af5c8e9c
I20250629 01:51:48.864722 451 catalog_manager.cc:6652] ReplaceTablet: tablet bfa48f94ac0e48e39578a6e8af5c8e9c deleted and replaced by tablet 11b4bdbbf5d3421ab6e3a31de90fff6d
I20250629 01:51:48.865015 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_218288faefcf4b5d9df5adbf5d4223cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.865484 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_218288faefcf4b5d9df5adbf5d4223cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.866533 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ea00b4559e03410885180286b5e9fd10
I20250629 01:51:48.866927 448 catalog_manager.cc:6652] ReplaceTablet: tablet ea00b4559e03410885180286b5e9fd10 deleted and replaced by tablet 34bf4ae5be0642da90503389d178ef4b
I20250629 01:51:48.867219 885 tablet_service.cc:1515] Processing DeleteTablet for tablet ea00b4559e03410885180286b5e9fd10 with delete_type TABLET_DATA_DELETED (replaced by tablet 34bf4ae5be0642da90503389d178ef4b) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:48.868978 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11b4bdbbf5d3421ab6e3a31de90fff6d has no consensus state
I20250629 01:51:48.870008 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f7d1b03023842daa7f9d43483dc6b95" } modify_external_catalogs: true
I20250629 01:51:48.870159 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f7d1b03023842daa7f9d43483dc6b95" } modify_external_catalogs: true
I20250629 01:51:48.870739 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cbbc85d10dc546b59c06dd6296eaa2db" } modify_external_catalogs: true
I20250629 01:51:48.870916 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cbbc85d10dc546b59c06dd6296eaa2db" } modify_external_catalogs: true
I20250629 01:51:48.874097 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_03045b3c97554436b5b29949fe6a2ccb" } new_table_name: "default.Table_2acf6246904d41ab94aad9dec7df2291" modify_external_catalogs: true
I20250629 01:51:48.877750 1155 tablet_bootstrap.cc:492] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.877889 1155 ts_tablet_manager.cc:1397] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.283s user 0.002s sys 0.008s
I20250629 01:51:48.878630 1155 raft_consensus.cc:357] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.878793 1155 raft_consensus.cc:383] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.878901 1155 raft_consensus.cc:738] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.879020 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 556) took 1847 ms. Trace:
I20250629 01:51:48.879138 1155 consensus_queue.cc:260] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.879141 1157 rpcz_store.cc:276] 0629 01:51:47.031101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.033064 (+ 1963us) service_pool.cc:225] Handling call
0629 01:51:48.549618 (+1516554us) ts_tablet_manager.cc:1145] Deleting tablet 0f40aefcb7e14612bf2d435dfa37d66e
0629 01:51:48.549627 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.574401 (+ 24774us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:48.858542 (+284141us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:48.879004 (+ 20462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490287}
I20250629 01:51:48.879892 1710 consensus_queue.cc:1035] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:48.880174 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7e9bf1a6fab54b3198412022c8662ec0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:48.880901 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 557) took 1677 ms. Trace:
I20250629 01:51:48.880609 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a726ca476374467fb904639d799b7a1c" } new_table_name: "default.Table_99e79a9aa55444b6b6d1873d0d4844c8" modify_external_catalogs: true
I20250629 01:51:48.880976 1157 rpcz_store.cc:276] 0629 01:51:47.203610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.203661 (+ 51us) service_pool.cc:225] Handling call
0629 01:51:48.880795 (+1677134us) ts_tablet_manager.cc:1145] Deleting tablet 432680cae61946af9859760fe23110ec
0629 01:51:48.880802 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.880886 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676953}
I20250629 01:51:48.881249 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 558) took 1677 ms. Trace:
I20250629 01:51:48.881319 1157 rpcz_store.cc:276] 0629 01:51:47.203968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.204007 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:48.881163 (+1677156us) ts_tablet_manager.cc:1145] Deleting tablet 50d23f589de84dd5b150a17cee00c411
0629 01:51:48.881170 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.881234 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676977}
I20250629 01:51:48.881564 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 559) took 1669 ms. Trace:
I20250629 01:51:48.881623 1157 rpcz_store.cc:276] 0629 01:51:47.212432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.213704 (+ 1272us) service_pool.cc:225] Handling call
0629 01:51:48.881486 (+1667782us) ts_tablet_manager.cc:1145] Deleting tablet e4d6f1eb43b24fda94aa0afcf4501195
0629 01:51:48.881493 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.881551 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667601}
I20250629 01:51:48.881758 1155 ts_tablet_manager.cc:1428] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:48.882002 1155 tablet_bootstrap.cc:492] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.882267 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 560) took 1517 ms. Trace:
I20250629 01:51:48.882340 1157 rpcz_store.cc:276] 0629 01:51:47.364798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.364842 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:48.881819 (+1516977us) ts_tablet_manager.cc:1145] Deleting tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d
0629 01:51:48.881825 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.882253 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516819}
W20250629 01:51:48.882858 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e9bf1a6fab54b3198412022c8662ec0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.882885 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 561) took 1513 ms. Trace:
I20250629 01:51:48.882947 1157 rpcz_store.cc:276] 0629 01:51:47.369717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.369831 (+ 114us) service_pool.cc:225] Handling call
0629 01:51:48.882807 (+1512976us) ts_tablet_manager.cc:1145] Deleting tablet e3b17172ab8b4e0dad36ebaac776dc03
0629 01:51:48.882813 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.882871 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1512812}
I20250629 01:51:48.883246 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 562) took 1504 ms. Trace:
I20250629 01:51:48.883087 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0f40aefcb7e14612bf2d435dfa37d66e (table default.Table_81df428605084d568b00885fe59d5e29 [id=db4ab9642af04dd8b4fe207346a42384]) successfully deleted
I20250629 01:51:48.883323 1157 rpcz_store.cc:276] 0629 01:51:47.378797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.378842 (+ 45us) service_pool.cc:225] Handling call
0629 01:51:48.883165 (+1504323us) ts_tablet_manager.cc:1145] Deleting tablet ec0ca04e5f6b42df8567deb53a3e5f53
0629 01:51:48.883172 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.883232 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1504176}
W20250629 01:51:48.883526 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 432680cae61946af9859760fe23110ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 432680cae61946af9859760fe23110ec already in progress: creating tablet
I20250629 01:51:48.883725 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 564) took 1468 ms. Trace:
W20250629 01:51:48.883718 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 50d23f589de84dd5b150a17cee00c411 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50d23f589de84dd5b150a17cee00c411 already in progress: creating tablet
I20250629 01:51:48.883788 1157 rpcz_store.cc:276] 0629 01:51:47.415142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.415189 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:48.883647 (+1468458us) ts_tablet_manager.cc:1145] Deleting tablet ef3060d46ac24330be49103920dc1dee
0629 01:51:48.883654 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.883712 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1468300}
W20250629 01:51:48.883905 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e4d6f1eb43b24fda94aa0afcf4501195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4d6f1eb43b24fda94aa0afcf4501195 already in progress: creating tablet
I20250629 01:51:48.883994 1157 tablet_replica.cc:331] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:48.884119 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d already in progress: creating tablet
I20250629 01:51:48.884192 1157 raft_consensus.cc:2241] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:48.884373 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e3b17172ab8b4e0dad36ebaac776dc03 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3b17172ab8b4e0dad36ebaac776dc03 already in progress: creating tablet
I20250629 01:51:48.884402 1157 raft_consensus.cc:2270] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:48.884791 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ec0ca04e5f6b42df8567deb53a3e5f53 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0ca04e5f6b42df8567deb53a3e5f53 already in progress: creating tablet
I20250629 01:51:48.884846 1157 ts_tablet_manager.cc:1905] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.884986 1155 tablet_bootstrap.cc:654] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:48.884994 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ef3060d46ac24330be49103920dc1dee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef3060d46ac24330be49103920dc1dee already in progress: creating tablet
I20250629 01:51:48.886334 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 5dfe921a84694903bbdb3fed3d6f2761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:48.887312 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet dbfcb9233ace429db5daa593c2a1dcbe (table default.Table_0247394d7c3c4941a7ef5ad5fa572566 [id=f05487296c954ad08343c2a49b1934ca], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dbfcb9233ace429db5daa593c2a1dcbe consensus state has no leader
I20250629 01:51:48.889580 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9624117e97974a47ac2bfe525da6f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.889706 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 28eaa55b3b564e2e9cf698fda98a4c8a (table default.Table_8b976eb77218429b9513d370b45cd5c9 [id=67701e562ee84eaf9030d915a726ebe2]) successfully deleted
I20250629 01:51:48.889909 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ea00b4559e03410885180286b5e9fd10 with delete_type TABLET_DATA_DELETED (replaced by tablet 34bf4ae5be0642da90503389d178ef4b) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:48.889925 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3070ce8c467a47b0a833374450911291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3070ce8c467a47b0a833374450911291 already in progress: creating tablet
I20250629 01:51:48.890100 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet bfa48f94ac0e48e39578a6e8af5c8e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 11b4bdbbf5d3421ab6e3a31de90fff6d) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:48.890151 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2249d5cba1544653b088312e7d74802e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2249d5cba1544653b088312e7d74802e already in progress: creating tablet
W20250629 01:51:48.890329 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0665d3f1225c44f894681fef4ae2520b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0665d3f1225c44f894681fef4ae2520b already in progress: creating tablet
W20250629 01:51:48.890554 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f3bdc8bda8a940be865679a05b90bd08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bdc8bda8a940be865679a05b90bd08 already in progress: creating tablet
W20250629 01:51:48.890753 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0827ec622ff04ca6b2fec50e40b9e1de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0827ec622ff04ca6b2fec50e40b9e1de already in progress: creating tablet
I20250629 01:51:48.891075 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ea00b4559e03410885180286b5e9fd10 with delete_type TABLET_DATA_DELETED (replaced by tablet 34bf4ae5be0642da90503389d178ef4b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.891410 743 tablet_service.cc:1515] Processing DeleteTablet for tablet bfa48f94ac0e48e39578a6e8af5c8e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 11b4bdbbf5d3421ab6e3a31de90fff6d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.891788 1156 tablet_replica.cc:331] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.892078 1156 raft_consensus.cc:2241] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.892364 1156 raft_consensus.cc:2270] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.892987 1156 ts_tablet_manager.cc:1905] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.895589 881 tablet_service.cc:1515] Processing DeleteTablet for tablet bfa48f94ac0e48e39578a6e8af5c8e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 11b4bdbbf5d3421ab6e3a31de90fff6d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.896793 1158 tablet_replica.cc:331] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.896962 1158 raft_consensus.cc:2241] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.897125 1158 raft_consensus.cc:2270] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.897605 1158 ts_tablet_manager.cc:1905] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.900856 1156 ts_tablet_manager.cc:1918] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:48.901021 1156 log.cc:1199] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/bfa48f94ac0e48e39578a6e8af5c8e9c
I20250629 01:51:48.901645 1156 ts_tablet_manager.cc:1939] T bfa48f94ac0e48e39578a6e8af5c8e9c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.907431 1159 tablet_bootstrap.cc:492] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:48.907562 1159 ts_tablet_manager.cc:1397] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.005s
I20250629 01:51:48.908288 1159 raft_consensus.cc:357] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.908449 1159 raft_consensus.cc:383] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.908517 1159 raft_consensus.cc:738] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.908727 1159 consensus_queue.cc:260] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:48.908928 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ea00b4559e03410885180286b5e9fd10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea00b4559e03410885180286b5e9fd10 already in progress: creating tablet
I20250629 01:51:48.909127 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]) successfully deleted
I20250629 01:51:48.906524 1155 tablet_bootstrap.cc:492] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.918488 1154 tablet_bootstrap.cc:492] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:48.920280 1154 ts_tablet_manager.cc:1397] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.308s user 0.006s sys 0.044s
I20250629 01:51:48.912293 1155 ts_tablet_manager.cc:1397] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250629 01:51:48.921054 1154 raft_consensus.cc:357] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.921324 1154 raft_consensus.cc:383] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.921424 1154 raft_consensus.cc:738] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.921656 1154 consensus_queue.cc:260] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.921883 1155 raft_consensus.cc:357] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.922091 1155 raft_consensus.cc:383] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.922175 1155 raft_consensus.cc:738] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.922619 1154 ts_tablet_manager.cc:1428] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:48.924525 1664 raft_consensus.cc:491] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.924670 1664 raft_consensus.cc:513] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.925215 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 82d58c88a6994d828b444c5814b287ff
I20250629 01:51:48.925230 1664 leader_election.cc:290] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.926508 1155 consensus_queue.cc:260] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.927525 1155 ts_tablet_manager.cc:1428] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:48.928256 1689 raft_consensus.cc:491] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.928351 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d58c88a6994d828b444c5814b287ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.928371 1689 raft_consensus.cc:513] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.928825 1689 leader_election.cc:290] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:48.929955 850 leader_election.cc:343] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:48.930140 1710 raft_consensus.cc:491] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.930274 1710 raft_consensus.cc:513] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.927734 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 881461aa1bc34dd483f65e171c341bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.928026 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d58c88a6994d828b444c5814b287ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.931291 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86832a3c26224c7ea0b9b4b6866d3880" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:48.931519 1156 tablet_replica.cc:331] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.931710 1156 raft_consensus.cc:2241] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.931859 1156 raft_consensus.cc:2270] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:48.931840 986 leader_election.cc:343] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.932493 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "86832a3c26224c7ea0b9b4b6866d3880" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.932559 1156 ts_tablet_manager.cc:1905] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:48.932921 988 leader_election.cc:343] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.933019 988 leader_election.cc:304] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.933233 1720 raft_consensus.cc:2747] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.933377 1710 leader_election.cc:290] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:48.933905 1159 ts_tablet_manager.cc:1428] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:51:48.934147 1159 tablet_bootstrap.cc:492] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:48.935038 1155 tablet_bootstrap.cc:492] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.936029 1159 tablet_bootstrap.cc:654] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.936492 1155 tablet_bootstrap.cc:654] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.936678 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.941614 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 82d58c88a6994d828b444c5814b287ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:48.942890 455 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 34bf4ae5be0642da90503389d178ef4b
W20250629 01:51:48.943243 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b consensus state has no leader
I20250629 01:51:48.943395 1158 ts_tablet_manager.cc:1918] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.943476 1158 log.cc:1199] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/351f4a7ae7e244999dda81d282de505d
I20250629 01:51:48.943778 1158 ts_tablet_manager.cc:1939] T 351f4a7ae7e244999dda81d282de505d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:48.943882 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd5150eff1ad43588a9e70dab47744e3 consensus state has no leader
I20250629 01:51:48.944677 1155 tablet_bootstrap.cc:492] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:48.944819 1155 ts_tablet_manager.cc:1397] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:48.945498 1155 raft_consensus.cc:357] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.945638 1155 raft_consensus.cc:383] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:48.945715 1155 raft_consensus.cc:738] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:48.945881 1155 consensus_queue.cc:260] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.946149 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4afb0343bde845f3810f908c9ffb78ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.946647 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4afb0343bde845f3810f908c9ffb78ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.948037 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bedbb041e4094d8399de625badd971bb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.950979 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d86465b0bdc64cf384232c87c1b67862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.951424 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d86465b0bdc64cf384232c87c1b67862 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:48.952957 712 leader_election.cc:343] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.954928 1157 ts_tablet_manager.cc:1918] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.955003 1157 log.cc:1199] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f1e0de66406e4ce89ce56fa447fe9895
I20250629 01:51:48.955220 1157 ts_tablet_manager.cc:1939] T f1e0de66406e4ce89ce56fa447fe9895 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:48.956499 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bedbb041e4094d8399de625badd971bb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:48.956653 1154 tablet_bootstrap.cc:492] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:48.956665 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_de40e2ec822b4c29a10c638c0a7d6f46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:48.957067 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de40e2ec822b4c29a10c638c0a7d6f46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:48.957921 1154 tablet_bootstrap.cc:654] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.956655 1048 raft_consensus.cc:2466] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:48.959518 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_541d81feb03a418780ca4f7da1ed1d4c" } modify_external_catalogs: true
I20250629 01:51:48.959703 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_541d81feb03a418780ca4f7da1ed1d4c" } modify_external_catalogs: true
I20250629 01:51:48.960430 1158 tablet_replica.cc:331] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:48.960613 1158 raft_consensus.cc:2241] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:48.960719 1158 raft_consensus.cc:2270] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.960682 710 leader_election.cc:304] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:51:48.961158 850 leader_election.cc:343] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:48.961241 850 leader_election.cc:304] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:48.961263 1158 ts_tablet_manager.cc:1905] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.963902 1156 ts_tablet_manager.cc:1918] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.964020 1156 log.cc:1199] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/881461aa1bc34dd483f65e171c341bc5
I20250629 01:51:48.964366 1156 ts_tablet_manager.cc:1939] T 881461aa1bc34dd483f65e171c341bc5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:48.966046 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:48.973552 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d61687e19d994e87b09e55bd3b9476d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:48.973750 1664 raft_consensus.cc:491] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:48.973892 1664 raft_consensus.cc:513] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.974520 1664 leader_election.cc:290] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.975831 1664 raft_consensus.cc:2802] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:48.975914 1664 raft_consensus.cc:491] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:48.975996 1664 raft_consensus.cc:3058] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.976634 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75fd4ffcf896419b81a1c0eb7c5aeeec" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:48.977030 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:48.978742 712 leader_election.cc:343] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.979357 1664 raft_consensus.cc:513] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.979598 1801 raft_consensus.cc:2747] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:48.979627 710 leader_election.cc:343] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 75fd4ffcf896419b81a1c0eb7c5aeeec
I20250629 01:51:48.980355 710 leader_election.cc:304] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:48.980618 1766 raft_consensus.cc:2747] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:48.980507 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bedbb041e4094d8399de625badd971bb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:48.981438 712 leader_election.cc:343] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:48.981456 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bedbb041e4094d8399de625badd971bb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:48.981580 1048 raft_consensus.cc:3058] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:48.981607 1664 leader_election.cc:290] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:48.982378 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 881461aa1bc34dd483f65e171c341bc5 (table default.Table_16db51abc1404d7fa67f1c00a70f9166 [id=a922c0f8e123451fb4060f61472e8d41]) successfully deleted
I20250629 01:51:48.982498 1155 ts_tablet_manager.cc:1428] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20250629 01:51:48.982699 1156 tablet_replica.cc:331] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:48.982849 1156 raft_consensus.cc:2241] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:48.983100 1156 raft_consensus.cc:2270] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:48.983568 1156 ts_tablet_manager.cc:1905] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:48.983860 1048 raft_consensus.cc:2466] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:48.984205 710 leader_election.cc:304] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:48.984578 1664 raft_consensus.cc:2802] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:48.984740 1664 raft_consensus.cc:695] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:48.984995 1664 consensus_queue.cc:237] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:48.988868 448 master_service.cc:948] ReplaceTablet: received request to replace tablet bd5150eff1ad43588a9e70dab47744e3 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:48.989077 1155 tablet_bootstrap.cc:492] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:48.990491 1155 tablet_bootstrap.cc:654] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:48.999199 1158 ts_tablet_manager.cc:1918] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:48.999298 1158 log.cc:1199] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:48.999562 1158 ts_tablet_manager.cc:1939] T 49333302a27f4ac3b76a1e492b3c1b60 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:49.002291 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 569) took 1565 ms. Trace:
I20250629 01:51:49.002372 1157 rpcz_store.cc:276] 0629 01:51:47.436973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.437009 (+ 36us) service_pool.cc:225] Handling call
0629 01:51:48.883960 (+1446951us) ts_tablet_manager.cc:1145] Deleting tablet f1e0de66406e4ce89ce56fa447fe9895
0629 01:51:48.883968 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:48.952458 (+ 68490us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:48.954868 (+ 2410us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:49.002274 (+ 47406us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5609,"tablet-delete.queue_time_us":1446780}
I20250629 01:51:49.002746 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f1e0de66406e4ce89ce56fa447fe9895 (table default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]) successfully deleted
I20250629 01:51:49.003261 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 572) took 1549 ms. Trace:
I20250629 01:51:49.003330 1157 rpcz_store.cc:276] 0629 01:51:47.453395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.453440 (+ 45us) service_pool.cc:225] Handling call
0629 01:51:49.003158 (+1549718us) ts_tablet_manager.cc:1145] Deleting tablet 56b9ef027c9d42048e4a6dff221df793
0629 01:51:49.003167 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.003246 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1549548}
I20250629 01:51:49.003603 1155 tablet_bootstrap.cc:492] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.003732 1155 ts_tablet_manager.cc:1397] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
W20250629 01:51:49.003706 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 56b9ef027c9d42048e4a6dff221df793 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56b9ef027c9d42048e4a6dff221df793 already in progress: creating tablet
I20250629 01:51:49.004033 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 574) took 1501 ms. Trace:
I20250629 01:51:49.004132 1157 rpcz_store.cc:276] 0629 01:51:47.503020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.503061 (+ 41us) service_pool.cc:225] Handling call
0629 01:51:49.003943 (+1500882us) ts_tablet_manager.cc:1145] Deleting tablet 8a880c9bb4b94d3b9b12f28d48565f06
0629 01:51:49.003952 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.004019 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1493366}
I20250629 01:51:49.004292 1664 raft_consensus.cc:491] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:49.004479 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8a880c9bb4b94d3b9b12f28d48565f06 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a880c9bb4b94d3b9b12f28d48565f06 already in progress: creating tablet
I20250629 01:51:49.004563 1664 raft_consensus.cc:513] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.004757 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 575) took 1496 ms. Trace:
I20250629 01:51:49.004822 1157 rpcz_store.cc:276] 0629 01:51:47.507855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.507900 (+ 45us) service_pool.cc:225] Handling call
0629 01:51:49.004669 (+1496769us) ts_tablet_manager.cc:1145] Deleting tablet 6ca02bcd9467496c9057b3de9c1b1725
0629 01:51:49.004677 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.004743 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492809}
W20250629 01:51:49.005162 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6ca02bcd9467496c9057b3de9c1b1725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ca02bcd9467496c9057b3de9c1b1725 already in progress: creating tablet
I20250629 01:51:49.005476 1157 tablet_replica.cc:331] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:49.005651 1157 raft_consensus.cc:2241] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.005774 1157 raft_consensus.cc:2270] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:49.006032 710 leader_election.cc:343] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d61687e19d994e87b09e55bd3b9476d7
I20250629 01:51:49.005761 1664 leader_election.cc:290] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:49.006282 1157 ts_tablet_manager.cc:1905] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.006397 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d61687e19d994e87b09e55bd3b9476d7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:49.006770 712 leader_election.cc:343] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.006893 712 leader_election.cc:304] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:49.007004 1808 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:49.007332 1664 raft_consensus.cc:2747] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.007645 1155 raft_consensus.cc:357] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:49.007799 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 45948.7us
I20250629 01:51:49.007835 1155 raft_consensus.cc:383] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.007889 1155 raft_consensus.cc:738] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.008085 1155 consensus_queue.cc:260] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.008692 1159 tablet_bootstrap.cc:492] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:49.008792 1159 ts_tablet_manager.cc:1397] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.000s
I20250629 01:51:49.009364 1159 raft_consensus.cc:357] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.009506 1159 raft_consensus.cc:383] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.009559 1159 raft_consensus.cc:738] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.009721 1159 consensus_queue.cc:260] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.011852 1155 ts_tablet_manager.cc:1428] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250629 01:51:49.013717 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.013964 1159 ts_tablet_manager.cc:1428] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:49.016633 1159 tablet_bootstrap.cc:492] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:49.018642 1155 tablet_bootstrap.cc:492] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.019793 1804 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:49.020392 1159 tablet_bootstrap.cc:654] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.021586 1156 ts_tablet_manager.cc:1918] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:49.021639 1154 tablet_bootstrap.cc:492] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.021755 1154 ts_tablet_manager.cc:1397] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.002s
I20250629 01:51:49.022310 1154 raft_consensus.cc:357] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.022435 1154 raft_consensus.cc:383] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.022493 1154 raft_consensus.cc:738] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.022661 1154 consensus_queue.cc:260] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.024811 1156 log.cc:1199] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/82d58c88a6994d828b444c5814b287ff
I20250629 01:51:49.025072 887 tablet_service.cc:1468] Processing CreateTablet for tablet 50e8d545cd234faaa20611e49724e17f (DEFAULT_TABLE table=default.Table_412fa53923654534afbfbdb488806ed2 [id=de55b091742d4261a27c49c6c2a78be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.025133 1154 ts_tablet_manager.cc:1428] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:49.023118 1155 tablet_bootstrap.cc:654] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.025384 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e8d545cd234faaa20611e49724e17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.025326 453 catalog_manager.cc:5582] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:49.025686 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:49.026876 1156 ts_tablet_manager.cc:1939] T 82d58c88a6994d828b444c5814b287ff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.027684 1156 tablet_replica.cc:331] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:49.027837 1156 raft_consensus.cc:2241] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.027969 1156 raft_consensus.cc:2270] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.031715 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 50e8d545cd234faaa20611e49724e17f (DEFAULT_TABLE table=default.Table_412fa53923654534afbfbdb488806ed2 [id=de55b091742d4261a27c49c6c2a78be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.032013 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e8d545cd234faaa20611e49724e17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.032888 1158 tablet_replica.cc:331] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:49.032889 887 tablet_service.cc:1468] Processing CreateTablet for tablet a94e804eb65845d8bc6ba96f87e97b47 (DEFAULT_TABLE table=default.Table_217838e5ec6344b8bfad7c93057cf5a4 [id=6c7ca52a6d254f109744b94922123d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.033023 1158 raft_consensus.cc:2241] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.033149 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94e804eb65845d8bc6ba96f87e97b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.033149 1158 raft_consensus.cc:2270] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.034821 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]) successfully deleted
I20250629 01:51:49.034914 743 tablet_service.cc:1468] Processing CreateTablet for tablet a94e804eb65845d8bc6ba96f87e97b47 (DEFAULT_TABLE table=default.Table_217838e5ec6344b8bfad7c93057cf5a4 [id=6c7ca52a6d254f109744b94922123d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.035216 749 tablet_service.cc:1468] Processing CreateTablet for tablet 50e8d545cd234faaa20611e49724e17f (DEFAULT_TABLE table=default.Table_412fa53923654534afbfbdb488806ed2 [id=de55b091742d4261a27c49c6c2a78be9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.036564 1157 ts_tablet_manager.cc:1918] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.036648 1157 log.cc:1199] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/3ba0e267f48a4bf1b58fb2339694719f
I20250629 01:51:49.036875 1157 ts_tablet_manager.cc:1939] T 3ba0e267f48a4bf1b58fb2339694719f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:49.036849 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fb0c134a9a1548cd8cebc3508bde937a" } new_table_name: "default.Table_9c12a09abc39412497235b71efb0ea8e" modify_external_catalogs: true
I20250629 01:51:49.037396 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 573) took 1537 ms. Trace:
I20250629 01:51:49.037456 1157 rpcz_store.cc:276] 0629 01:51:47.499901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.505155 (+ 5254us) service_pool.cc:225] Handling call
0629 01:51:49.005436 (+1500281us) ts_tablet_manager.cc:1145] Deleting tablet 3ba0e267f48a4bf1b58fb2339694719f
0629 01:51:49.005445 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.034156 (+ 28711us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:49.036503 (+ 2347us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:49.037379 (+ 876us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1480898}
I20250629 01:51:49.037781 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 3ba0e267f48a4bf1b58fb2339694719f (table default.Table_3abcbdb38a38484782a28269e7b69beb [id=29b0e64154664e928993c1035a1bb23c]) successfully deleted
I20250629 01:51:49.038064 1154 tablet_bootstrap.cc:492] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.038050 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 49333302a27f4ac3b76a1e492b3c1b60 (table default.Table_3ca4f32bc0954a7c92345faedf91b471 [id=48089fc1034648639f16290cb405456a]) successfully deleted
W20250629 01:51:49.038239 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 897a4cefefe04f06b47070ec7fa1a33b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897a4cefefe04f06b47070ec7fa1a33b already in progress: creating tablet
W20250629 01:51:49.038468 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4d9ef77181ba48f3a6de16b945c8df1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d9ef77181ba48f3a6de16b945c8df1a already in progress: creating tablet
W20250629 01:51:49.038646 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b75d33594c994363bdb2558eb0c6ca3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b75d33594c994363bdb2558eb0c6ca3d already in progress: creating tablet
W20250629 01:51:49.038856 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 258994dd24804f4aaa5e299436fac9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 258994dd24804f4aaa5e299436fac9b0 already in progress: creating tablet
W20250629 01:51:49.039027 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 871086e50583453cadcc60ccd8afcba6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871086e50583453cadcc60ccd8afcba6 already in progress: creating tablet
I20250629 01:51:49.039351 1154 tablet_bootstrap.cc:654] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.040772 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 17acb9bcd7aa4a508d1695474a1b56df
I20250629 01:51:49.041360 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 17acb9bcd7aa4a508d1695474a1b56df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.041496 455 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 543242e4a5064268ae1e5519b569f682 (table default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]). Expected version 1 got 0
I20250629 01:51:49.035218 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94e804eb65845d8bc6ba96f87e97b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.042877 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50e8d545cd234faaa20611e49724e17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.044291 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 52be184d0480495a894442d6f72a0f12
I20250629 01:51:49.044277 881 tablet_service.cc:1468] Processing CreateTablet for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 (DEFAULT_TABLE table=default.Table_13dc4660837b46219b404433b7272fce [id=ece0dac89dd34051bd2f1163dd3c9bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.044539 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84cfa2d73c6b4ca2bf41437b196d10b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.046437 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 17acb9bcd7aa4a508d1695474a1b56df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.046766 747 tablet_service.cc:1468] Processing CreateTablet for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 (DEFAULT_TABLE table=default.Table_13dc4660837b46219b404433b7272fce [id=ece0dac89dd34051bd2f1163dd3c9bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.047111 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84cfa2d73c6b4ca2bf41437b196d10b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.048414 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 52be184d0480495a894442d6f72a0f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.048650 885 tablet_service.cc:1468] Processing CreateTablet for tablet 59cc1690878f40e39e23e1942c75effa (DEFAULT_TABLE table=default.Table_620b12abd6f9476e8bf5127f618d9295 [id=688cfc291a34490ba631b108fe6f05c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.048887 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59cc1690878f40e39e23e1942c75effa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.051918 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 52be184d0480495a894442d6f72a0f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.052227 757 tablet_service.cc:1468] Processing CreateTablet for tablet 59cc1690878f40e39e23e1942c75effa (DEFAULT_TABLE table=default.Table_620b12abd6f9476e8bf5127f618d9295 [id=688cfc291a34490ba631b108fe6f05c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.052433 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6b4d4d6dacb84f80903522d20de4e795 (DEFAULT_TABLE table=default.Table_f4cdf270acad4465adbb179a26cedb39 [id=5d74773751e44e069d953c374259e5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.052549 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bd5150eff1ad43588a9e70dab47744e3
I20250629 01:51:49.052667 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4d4d6dacb84f80903522d20de4e795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.052798 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 17acb9bcd7aa4a508d1695474a1b56df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.052915 448 catalog_manager.cc:6652] ReplaceTablet: tablet bd5150eff1ad43588a9e70dab47744e3 deleted and replaced by tablet 0bef2747f0c04f498079906e40bac24c
I20250629 01:51:49.053117 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 (DEFAULT_TABLE table=default.Table_13dc4660837b46219b404433b7272fce [id=ece0dac89dd34051bd2f1163dd3c9bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.053403 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84cfa2d73c6b4ca2bf41437b196d10b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.054826 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d499c5244b63442cbb0110787b2c209d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.055296 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d499c5244b63442cbb0110787b2c209d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.056171 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 52be184d0480495a894442d6f72a0f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.056448 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 59cc1690878f40e39e23e1942c75effa (DEFAULT_TABLE table=default.Table_620b12abd6f9476e8bf5127f618d9295 [id=688cfc291a34490ba631b108fe6f05c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.056706 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59cc1690878f40e39e23e1942c75effa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.058230 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 77eae224763d464496066b4a98efe5ff
I20250629 01:51:49.059093 877 tablet_service.cc:1468] Processing CreateTablet for tablet 22567416bfde4e2b843cf5c7a214ec6c (DEFAULT_TABLE table=default.Table_59e6d6754ff145b8bea3842035e15189 [id=3fdeb1ad0a864b98a54efba45dcdb98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.059369 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22567416bfde4e2b843cf5c7a214ec6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.059489 752 tablet_service.cc:1468] Processing CreateTablet for tablet 22567416bfde4e2b843cf5c7a214ec6c (DEFAULT_TABLE table=default.Table_59e6d6754ff145b8bea3842035e15189 [id=3fdeb1ad0a864b98a54efba45dcdb98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.059804 893 tablet_service.cc:1468] Processing CreateTablet for tablet ab59d0a03543499fbacaf39743ca812b (DEFAULT_TABLE table=default.Table_28e044c713374ff284d41c0262e1bbc2 [id=5719fc86f39d4e4d9f2bbf4038c6c0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.059835 751 tablet_service.cc:1468] Processing CreateTablet for tablet bae91d28f27f4e14a23c65e41091b671 (DEFAULT_TABLE table=default.Table_2c656b7d46064f7d811de6d03aa4224e [id=c968db78da2c44a6900dc90d8c395c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.060055 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59d0a03543499fbacaf39743ca812b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.060045 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 77eae224763d464496066b4a98efe5ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.060407 750 tablet_service.cc:1468] Processing CreateTablet for tablet 6b4d4d6dacb84f80903522d20de4e795 (DEFAULT_TABLE table=default.Table_f4cdf270acad4465adbb179a26cedb39 [id=5d74773751e44e069d953c374259e5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.060653 742 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5150eff1ad43588a9e70dab47744e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bef2747f0c04f498079906e40bac24c) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.061215 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 22567416bfde4e2b843cf5c7a214ec6c (DEFAULT_TABLE table=default.Table_59e6d6754ff145b8bea3842035e15189 [id=3fdeb1ad0a864b98a54efba45dcdb98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.061481 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22567416bfde4e2b843cf5c7a214ec6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.059089 881 tablet_service.cc:1468] Processing CreateTablet for tablet bae91d28f27f4e14a23c65e41091b671 (DEFAULT_TABLE table=default.Table_2c656b7d46064f7d811de6d03aa4224e [id=c968db78da2c44a6900dc90d8c395c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.062093 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae91d28f27f4e14a23c65e41091b671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.064021 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5150eff1ad43588a9e70dab47744e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bef2747f0c04f498079906e40bac24c) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.064289 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 6b4d4d6dacb84f80903522d20de4e795 (DEFAULT_TABLE table=default.Table_f4cdf270acad4465adbb179a26cedb39 [id=5d74773751e44e069d953c374259e5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.064563 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4d4d6dacb84f80903522d20de4e795. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.065802 988 connection.cc:537] client connection to 127.0.104.61:35873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250629 01:51:49.065990 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0ae8ed0950bd490598121af26c1d0eeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.066416 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ae8ed0950bd490598121af26c1d0eeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:49.066607 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6b6d5ff92c343cb8f1b3039b84390af consensus state has no leader
I20250629 01:51:49.066885 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_2657fd4cf2044332bcbf4e1027d13bee [id=d02cba06b5bc499280a223f49faa9e40] alter complete (version 3)
I20250629 01:51:49.067798 742 tablet_service.cc:1468] Processing CreateTablet for tablet ab59d0a03543499fbacaf39743ca812b (DEFAULT_TABLE table=default.Table_28e044c713374ff284d41c0262e1bbc2 [id=5719fc86f39d4e4d9f2bbf4038c6c0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.068950 1021 tablet_service.cc:1468] Processing CreateTablet for tablet bae91d28f27f4e14a23c65e41091b671 (DEFAULT_TABLE table=default.Table_2c656b7d46064f7d811de6d03aa4224e [id=c968db78da2c44a6900dc90d8c395c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.069235 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae91d28f27f4e14a23c65e41091b671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.070952 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7ca2e46142c5424ab8f471c13d635ca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.071399 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ca2e46142c5424ab8f471c13d635ca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.071784 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 77eae224763d464496066b4a98efe5ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.072181 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 77eae224763d464496066b4a98efe5ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.072371 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4799f303eeb041edb1117ae3caa45b62" } new_table_name: "default.Table_4f1e7e5479c84d75868d25a5c2745c88" modify_external_catalogs: true
I20250629 01:51:49.074625 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e9c328545beb404bb1a6535d809fd023" } new_table_name: "default.Table_b172df6df0ab4b738c77a9835b3f88c2" modify_external_catalogs: true
W20250629 01:51:49.081462 852 connection.cc:537] client connection to 127.0.104.61:35873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250629 01:51:49.081545 712 connection.cc:537] client connection to 127.0.104.61:35873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250629 01:51:49.081871 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.61:35873 (error 108) [suppressed 7 similar messages]
I20250629 01:51:49.082587 887 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5150eff1ad43588a9e70dab47744e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bef2747f0c04f498079906e40bac24c) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.082844 449 master_service.cc:948] ReplaceTablet: received request to replace tablet ca13eb6bdcd044ec948ea699857a30f8 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:49.083428 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59cc1690878f40e39e23e1942c75effa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.089761 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c799a04c0f74492b9fb674f9b0da240 consensus state has no leader
I20250629 01:51:49.093837 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ca13eb6bdcd044ec948ea699857a30f8
I20250629 01:51:49.094228 449 catalog_manager.cc:6652] ReplaceTablet: tablet ca13eb6bdcd044ec948ea699857a30f8 deleted and replaced by tablet 4234c00081e847e9ba6bb70e630ba764
I20250629 01:51:49.094758 757 tablet_service.cc:1515] Processing DeleteTablet for tablet ca13eb6bdcd044ec948ea699857a30f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4234c00081e847e9ba6bb70e630ba764) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:49.094897 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee7fb74854db4034a9d44f6d025e6756 consensus state has no leader
I20250629 01:51:49.096223 1158 ts_tablet_manager.cc:1905] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:49.096223 1816 negotiation.cc:337] Failed RPC negotiation. Trace:
0629 01:51:49.066638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:35873 (local address 127.0.104.3:47963)
0629 01:51:49.092799 (+ 26161us) negotiation.cc:107] Waiting for socket to connect
0629 01:51:49.092821 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0629 01:51:49.092943 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0629 01:51:49.095894 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":26037,"thread_start_us":28228,"threads_started":1}
W20250629 01:51:49.097121 1819 negotiation.cc:337] Failed RPC negotiation. Trace:
0629 01:51:49.082805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:35873 (local address 127.0.104.1:50229)
0629 01:51:49.095262 (+ 12457us) negotiation.cc:107] Waiting for socket to connect
0629 01:51:49.095280 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0629 01:51:49.095390 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0629 01:51:49.096751 (+ 1361us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12321,"thread_start_us":11951,"threads_started":1}
I20250629 01:51:49.100490 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet ca13eb6bdcd044ec948ea699857a30f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4234c00081e847e9ba6bb70e630ba764) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.100656 1159 tablet_bootstrap.cc:492] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:49.100939 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae91d28f27f4e14a23c65e41091b671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.101115 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a94e804eb65845d8bc6ba96f87e97b47 (DEFAULT_TABLE table=default.Table_217838e5ec6344b8bfad7c93057cf5a4 [id=6c7ca52a6d254f109744b94922123d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.101439 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a94e804eb65845d8bc6ba96f87e97b47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.104291 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22567416bfde4e2b843cf5c7a214ec6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.105645 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 80c9d1a365ee4c9992452bfb2de155a6 (DEFAULT_TABLE table=default.Table_b822effa18c64fe28fcfc3ff8db8c6f2 [id=fa619ffd472343b7b047e160bb112e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.105775 1154 tablet_bootstrap.cc:492] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.105906 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c9d1a365ee4c9992452bfb2de155a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.105893 1154 ts_tablet_manager.cc:1397] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.001s
I20250629 01:51:49.106917 1154 raft_consensus.cc:357] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.107105 1154 raft_consensus.cc:383] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.107159 1154 raft_consensus.cc:738] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.107375 1154 consensus_queue.cc:260] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.108018 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b4d4d6dacb84f80903522d20de4e795. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.108039 754 tablet_service.cc:1468] Processing CreateTablet for tablet f034e95729c040759534d36cd08f9ec9 (DEFAULT_TABLE table=default.Table_c17d83489bb0493ab307c56ab6212e37 [id=732eda07bef04e55a8b6d1effe89cd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.108361 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f034e95729c040759534d36cd08f9ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.111896 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59d0a03543499fbacaf39743ca812b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.118399 749 tablet_service.cc:1468] Processing CreateTablet for tablet ee7fb74854db4034a9d44f6d025e6756 (DEFAULT_TABLE table=default.Table_b172df6df0ab4b738c77a9835b3f88c2 [id=17372644e9864476a43de449608257ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.118687 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7fb74854db4034a9d44f6d025e6756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.118775 1025 tablet_service.cc:1468] Processing CreateTablet for tablet ee7fb74854db4034a9d44f6d025e6756 (DEFAULT_TABLE table=default.Table_b172df6df0ab4b738c77a9835b3f88c2 [id=17372644e9864476a43de449608257ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.119021 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7fb74854db4034a9d44f6d025e6756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.119200 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 7ecf8b55c83943b6b0b3fe2b5dcf87ad (DEFAULT_TABLE table=default.Table_eae63233f14743f6ba4f11140b92f156 [id=b44bf077efae48b7b965ed57a1198c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.119458 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf8b55c83943b6b0b3fe2b5dcf87ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.120282 1015 tablet_service.cc:1468] Processing CreateTablet for tablet ab59d0a03543499fbacaf39743ca812b (DEFAULT_TABLE table=default.Table_28e044c713374ff284d41c0262e1bbc2 [id=5719fc86f39d4e4d9f2bbf4038c6c0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.120537 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab59d0a03543499fbacaf39743ca812b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.123580 757 tablet_service.cc:1468] Processing CreateTablet for tablet 80c9d1a365ee4c9992452bfb2de155a6 (DEFAULT_TABLE table=default.Table_b822effa18c64fe28fcfc3ff8db8c6f2 [id=fa619ffd472343b7b047e160bb112e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.123582 1022 tablet_service.cc:1468] Processing CreateTablet for tablet f034e95729c040759534d36cd08f9ec9 (DEFAULT_TABLE table=default.Table_c17d83489bb0493ab307c56ab6212e37 [id=732eda07bef04e55a8b6d1effe89cd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.123837 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f034e95729c040759534d36cd08f9ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.123852 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c9d1a365ee4c9992452bfb2de155a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.119014 1154 ts_tablet_manager.cc:1428] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250629 01:51:49.126724 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99703cee14e34f2ebe6028b072a5b072" } modify_external_catalogs: true
I20250629 01:51:49.126885 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99703cee14e34f2ebe6028b072a5b072" } modify_external_catalogs: true
I20250629 01:51:49.130784 1154 tablet_bootstrap.cc:492] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.132056 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_34b52be75f5d4681945cc00f5911c2d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.132608 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34b52be75f5d4681945cc00f5911c2d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.133633 1154 tablet_bootstrap.cc:654] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.136541 747 tablet_service.cc:1468] Processing CreateTablet for tablet 7ecf8b55c83943b6b0b3fe2b5dcf87ad (DEFAULT_TABLE table=default.Table_eae63233f14743f6ba4f11140b92f156 [id=b44bf077efae48b7b965ed57a1198c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.136834 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf8b55c83943b6b0b3fe2b5dcf87ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.140372 742 tablet_service.cc:1468] Processing CreateTablet for tablet b5c32f8431be4eda8c77641c20ad4d1e (DEFAULT_TABLE table=default.Table_4de24a8e2d8f4c888ab0adb848c6deed [id=191fc7dbd60f44adba307484f6d6c745]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.140667 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c32f8431be4eda8c77641c20ad4d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.147959 751 tablet_service.cc:1468] Processing CreateTablet for tablet f52937830a924d6d8338d78da24b84cd (DEFAULT_TABLE table=default.Table_f7db52cd26e34c54b5fa3ba338525f24 [id=1089fcca80684ab5ba749f4251f246cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.148321 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52937830a924d6d8338d78da24b84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.149259 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7bb26e59c27447b09a8cacbad6133b3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.149680 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bb26e59c27447b09a8cacbad6133b3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.150666 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3fc2c8b5061f4a6e8f46cd439cf5c776" } modify_external_catalogs: true
I20250629 01:51:49.150806 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3fc2c8b5061f4a6e8f46cd439cf5c776" } modify_external_catalogs: true
I20250629 01:51:49.151846 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_caef1729000a42b482b4da2dcd58f39a" } new_table_name: "default.Table_8b1d03de93bf48a1a4f8ed74a7ac56c3" modify_external_catalogs: true
I20250629 01:51:49.153859 1025 tablet_service.cc:1468] Processing CreateTablet for tablet f52937830a924d6d8338d78da24b84cd (DEFAULT_TABLE table=default.Table_f7db52cd26e34c54b5fa3ba338525f24 [id=1089fcca80684ab5ba749f4251f246cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.154126 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52937830a924d6d8338d78da24b84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.154757 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 60e12d28ef784d898dbf9846f7063fdf (DEFAULT_TABLE table=default.Table_91cf07f495b64649afe5b6d70efa7dba [id=da9858daaef34614b39b51b0231ad0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.155015 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e12d28ef784d898dbf9846f7063fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.155558 752 tablet_service.cc:1468] Processing CreateTablet for tablet 60e12d28ef784d898dbf9846f7063fdf (DEFAULT_TABLE table=default.Table_91cf07f495b64649afe5b6d70efa7dba [id=da9858daaef34614b39b51b0231ad0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.155831 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e12d28ef784d898dbf9846f7063fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.156298 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3dd8aa3c2e914f019c0e7d29a3dbda13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.156685 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dd8aa3c2e914f019c0e7d29a3dbda13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.157577 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ea4ec96134d342ad8386604b1305651d" } new_table_name: "default.Table_39cde32ddcd642cc8b9240f4d3f979a6" modify_external_catalogs: true
I20250629 01:51:49.160223 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0c86a89fdf674dfc856f5a107daad436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.160600 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c86a89fdf674dfc856f5a107daad436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.163620 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f94d8fc1b6254ba69acadcaff3613d97
W20250629 01:51:49.163858 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfb025535ecd4cf88681c4951a56ba29 consensus state has no leader
I20250629 01:51:49.164803 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.164961 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3bdc6a26b3594d6fb33a27cac5b29f4f
I20250629 01:51:49.100766 1159 ts_tablet_manager.cc:1397] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
I20250629 01:51:49.166965 1159 raft_consensus.cc:357] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.167126 1159 raft_consensus.cc:383] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.167176 1159 raft_consensus.cc:738] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.167634 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ca02bcd9467496c9057b3de9c1b1725 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:49.163425 914 raft_consensus.cc:3058] T 955a41750b6a4766bb4fd3f01cfd99c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:49.125725 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 581) took 1449 ms. Trace:
I20250629 01:51:49.168715 1157 rpcz_store.cc:276] 0629 01:51:47.676144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.676183 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:49.125586 (+1449403us) ts_tablet_manager.cc:1145] Deleting tablet d7cbc2e6148846f9a58239b089e5a052
0629 01:51:49.125597 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.125704 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449183}
I20250629 01:51:49.169143 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 580) took 1493 ms. Trace:
I20250629 01:51:49.169203 1157 rpcz_store.cc:276] 0629 01:51:47.675672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.679997 (+ 4325us) service_pool.cc:225] Handling call
0629 01:51:49.169038 (+1489041us) ts_tablet_manager.cc:1145] Deleting tablet a40c1a8eec9549a8bdf695d6516440a8
0629 01:51:49.169046 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.169131 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488859}
I20250629 01:51:49.169544 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 579) took 1496 ms. Trace:
I20250629 01:51:49.169600 1157 rpcz_store.cc:276] 0629 01:51:47.672769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.680210 (+ 7441us) service_pool.cc:225] Handling call
0629 01:51:49.169462 (+1489252us) ts_tablet_manager.cc:1145] Deleting tablet 56b60431b72e4792aa1c32fdee40cdb9
0629 01:51:49.169468 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.169533 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1488566}
I20250629 01:51:49.169917 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 582) took 1487 ms. Trace:
I20250629 01:51:49.169967 1157 rpcz_store.cc:276] 0629 01:51:47.682358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.766538 (+ 84180us) service_pool.cc:225] Handling call
0629 01:51:49.169848 (+1403310us) ts_tablet_manager.cc:1145] Deleting tablet 1ea681bd91e54e4898bcfb72bed6198f
0629 01:51:49.169854 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.169907 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394070}
I20250629 01:51:49.170297 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 597) took 1338 ms. Trace:
I20250629 01:51:49.170352 1157 rpcz_store.cc:276] 0629 01:51:47.832276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.832316 (+ 40us) service_pool.cc:225] Handling call
0629 01:51:49.170221 (+1337905us) ts_tablet_manager.cc:1145] Deleting tablet 959837d39a9a4a6f8735a906b3a53222
0629 01:51:49.170228 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.170286 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337443}
I20250629 01:51:49.170490 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.170653 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 598) took 1202 ms. Trace:
I20250629 01:51:49.170709 1157 rpcz_store.cc:276] 0629 01:51:47.968037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.969585 (+ 1548us) service_pool.cc:225] Handling call
0629 01:51:49.170583 (+1200998us) ts_tablet_manager.cc:1145] Deleting tablet 67e90a48af8f48bba3b3fd4f4110d367
0629 01:51:49.170590 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.170643 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200405}
I20250629 01:51:49.171036 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 599) took 1202 ms. Trace:
I20250629 01:51:49.171099 1157 rpcz_store.cc:276] 0629 01:51:47.968175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:47.969633 (+ 1458us) service_pool.cc:225] Handling call
0629 01:51:49.170956 (+1201323us) ts_tablet_manager.cc:1145] Deleting tablet bccdcd9de08648bb925db6fe56f97d20
0629 01:51:49.170962 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.171025 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1192519}
W20250629 01:51:49.172791 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3874404c3444d47a71f1dce7fb86ff8 consensus state has no leader
I20250629 01:51:49.173579 1157 tablet_replica.cc:331] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:49.173956 1157 raft_consensus.cc:2241] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.173769 1664 consensus_queue.cc:1035] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:49.174090 1157 raft_consensus.cc:2270] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.174520 1157 ts_tablet_manager.cc:1905] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.178290 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.167323 1159 consensus_queue.cc:260] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.179129 1159 ts_tablet_manager.cc:1428] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250629 01:51:49.179412 1159 tablet_bootstrap.cc:492] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:49.181224 1159 tablet_bootstrap.cc:654] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.182364 747 tablet_service.cc:1515] Processing DeleteTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.184579 1032 tablet_service.cc:1468] Processing CreateTablet for tablet b5c32f8431be4eda8c77641c20ad4d1e (DEFAULT_TABLE table=default.Table_4de24a8e2d8f4c888ab0adb848c6deed [id=191fc7dbd60f44adba307484f6d6c745]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.186107 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e1fe3880e51345269e852014d0d60bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.186431 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c32f8431be4eda8c77641c20ad4d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.188748 1154 tablet_bootstrap.cc:492] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.188930 1154 ts_tablet_manager.cc:1397] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.005s
I20250629 01:51:49.189285 1158 ts_tablet_manager.cc:1918] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.189385 1158 log.cc:1199] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9624117e97974a47ac2bfe525da6f405
I20250629 01:51:49.189642 1158 ts_tablet_manager.cc:1939] T 9624117e97974a47ac2bfe525da6f405 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:49.189543 1154 raft_consensus.cc:357] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.189721 1154 raft_consensus.cc:383] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.189774 1154 raft_consensus.cc:738] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.189749 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_415759429f02467f944d3c1a215d5966"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.189992 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35873, user_credentials={real_user=slave}} blocked reactor thread for 108100us
W20250629 01:51:49.190153 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_415759429f02467f944d3c1a215d5966 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.190570 881 tablet_service.cc:1515] Processing DeleteTablet for tablet ca13eb6bdcd044ec948ea699857a30f8 with delete_type TABLET_DATA_DELETED (replaced by tablet 4234c00081e847e9ba6bb70e630ba764) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.190958 881 tablet_service.cc:1468] Processing CreateTablet for tablet 7ecf8b55c83943b6b0b3fe2b5dcf87ad (DEFAULT_TABLE table=default.Table_eae63233f14743f6ba4f11140b92f156 [id=b44bf077efae48b7b965ed57a1198c32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.191172 887 tablet_service.cc:1468] Processing CreateTablet for tablet 80c9d1a365ee4c9992452bfb2de155a6 (DEFAULT_TABLE table=default.Table_b822effa18c64fe28fcfc3ff8db8c6f2 [id=fa619ffd472343b7b047e160bb112e95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.191277 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ecf8b55c83943b6b0b3fe2b5dcf87ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.191447 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c9d1a365ee4c9992452bfb2de155a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.191509 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_61af8dc03a3049d59036eab5d6240298"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.191926 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61af8dc03a3049d59036eab5d6240298 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.192929 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3ec720103be44b36a6f7d15307a437ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.193390 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ec720103be44b36a6f7d15307a437ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.196580 1158 tablet_replica.cc:331] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:49.197865 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9624117e97974a47ac2bfe525da6f405 (table default.Table_3906fb981a8640e1906a78a321c479b2 [id=e2a731484fe346738ed611d5d7312092]) successfully deleted
I20250629 01:51:49.198324 1158 raft_consensus.cc:2241] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:49.198678 1158 raft_consensus.cc:2270] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.199311 1158 ts_tablet_manager.cc:1905] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.199585 881 tablet_service.cc:1468] Processing CreateTablet for tablet ee7fb74854db4034a9d44f6d025e6756 (DEFAULT_TABLE table=default.Table_b172df6df0ab4b738c77a9835b3f88c2 [id=17372644e9864476a43de449608257ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.199887 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7fb74854db4034a9d44f6d025e6756. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.200173 887 tablet_service.cc:1468] Processing CreateTablet for tablet f034e95729c040759534d36cd08f9ec9 (DEFAULT_TABLE table=default.Table_c17d83489bb0493ab307c56ab6212e37 [id=732eda07bef04e55a8b6d1effe89cd60]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.200441 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f034e95729c040759534d36cd08f9ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.200621 1157 ts_tablet_manager.cc:1918] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.200706 1157 log.cc:1199] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/36fcb819c61c4fabbb0da295d0002998
I20250629 01:51:49.200968 1157 ts_tablet_manager.cc:1939] T 36fcb819c61c4fabbb0da295d0002998 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:49.204522 877 tablet_service.cc:1468] Processing CreateTablet for tablet b5c32f8431be4eda8c77641c20ad4d1e (DEFAULT_TABLE table=default.Table_4de24a8e2d8f4c888ab0adb848c6deed [id=191fc7dbd60f44adba307484f6d6c745]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.204795 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c32f8431be4eda8c77641c20ad4d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.205862 885 tablet_service.cc:1468] Processing CreateTablet for tablet f52937830a924d6d8338d78da24b84cd (DEFAULT_TABLE table=default.Table_f7db52cd26e34c54b5fa3ba338525f24 [id=1089fcca80684ab5ba749f4251f246cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.206099 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52937830a924d6d8338d78da24b84cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.211092 885 tablet_service.cc:1468] Processing CreateTablet for tablet 60e12d28ef784d898dbf9846f7063fdf (DEFAULT_TABLE table=default.Table_91cf07f495b64649afe5b6d70efa7dba [id=da9858daaef34614b39b51b0231ad0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.211403 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60e12d28ef784d898dbf9846f7063fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.212839 1159 tablet_bootstrap.cc:492] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:49.212966 1159 ts_tablet_manager.cc:1397] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250629 01:51:49.213721 1159 raft_consensus.cc:357] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.213884 1159 raft_consensus.cc:383] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.213944 1159 raft_consensus.cc:738] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.214133 1159 consensus_queue.cc:260] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.214558 1159 ts_tablet_manager.cc:1428] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:49.214871 1159 tablet_bootstrap.cc:492] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:49.217240 1157 tablet_replica.cc:331] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:49.217415 1157 raft_consensus.cc:2241] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.217555 1157 raft_consensus.cc:2270] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.218225 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:49.218279 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d7cbc2e6148846f9a58239b089e5a052 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7cbc2e6148846f9a58239b089e5a052 already in progress: creating tablet
I20250629 01:51:49.218443 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:49.218480 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a40c1a8eec9549a8bdf695d6516440a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a40c1a8eec9549a8bdf695d6516440a8 already in progress: creating tablet
W20250629 01:51:49.218672 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 56b60431b72e4792aa1c32fdee40cdb9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56b60431b72e4792aa1c32fdee40cdb9 already in progress: creating tablet
W20250629 01:51:49.218845 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1ea681bd91e54e4898bcfb72bed6198f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ea681bd91e54e4898bcfb72bed6198f already in progress: creating tablet
W20250629 01:51:49.219034 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 959837d39a9a4a6f8735a906b3a53222 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 959837d39a9a4a6f8735a906b3a53222 already in progress: creating tablet
W20250629 01:51:49.219198 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 67e90a48af8f48bba3b3fd4f4110d367 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67e90a48af8f48bba3b3fd4f4110d367 already in progress: creating tablet
W20250629 01:51:49.219394 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet bccdcd9de08648bb925db6fe56f97d20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bccdcd9de08648bb925db6fe56f97d20 already in progress: creating tablet
W20250629 01:51:49.219556 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cb7314fbdfc74b1c8d9d1be9c05244cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7314fbdfc74b1c8d9d1be9c05244cb already in progress: creating tablet
W20250629 01:51:49.219738 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b6d21dfd48ea4a95a2fcecde6735974b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d21dfd48ea4a95a2fcecde6735974b already in progress: creating tablet
W20250629 01:51:49.219894 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 already in progress: creating tablet
I20250629 01:51:49.220281 1159 tablet_bootstrap.cc:654] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.220676 1154 consensus_queue.cc:260] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.221149 1154 ts_tablet_manager.cc:1428] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250629 01:51:49.221328 1157 ts_tablet_manager.cc:1905] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.221767 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 36fcb819c61c4fabbb0da295d0002998 (table default.Table_654a8089dcba4f64926b8b405150ffa5 [id=052b54452f124ba79d77e66ee2d68b90]) successfully deleted
I20250629 01:51:49.222061 1154 tablet_bootstrap.cc:492] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.224746 1154 tablet_bootstrap.cc:654] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.227792 1158 ts_tablet_manager.cc:1918] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:49.227881 1158 log.cc:1199] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/bfa48f94ac0e48e39578a6e8af5c8e9c
I20250629 01:51:49.228242 1158 ts_tablet_manager.cc:1939] T bfa48f94ac0e48e39578a6e8af5c8e9c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:49.229049 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]) successfully deleted
W20250629 01:51:49.229171 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 689adaf0c7564d9ba0b31061e4e2d8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 689adaf0c7564d9ba0b31061e4e2d8cf already in progress: creating tablet
W20250629 01:51:49.229391 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 119248ade66640299a7f17fb158a3a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 119248ade66640299a7f17fb158a3a5a already in progress: creating tablet
W20250629 01:51:49.231006 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 021da4395e414413b9b11ebcfbffcd19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 021da4395e414413b9b11ebcfbffcd19 already in progress: creating tablet
I20250629 01:51:49.232549 1157 ts_tablet_manager.cc:1918] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.232640 1157 log.cc:1199] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/af9ab324ce7c488cbaf6e0f0322b5c58
I20250629 01:51:49.232890 1157 ts_tablet_manager.cc:1939] T af9ab324ce7c488cbaf6e0f0322b5c58 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:49.233861 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.234851 1158 tablet_replica.cc:331] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:49.235165 1158 raft_consensus.cc:2241] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.235510 1158 raft_consensus.cc:2270] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.235569 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet af9ab324ce7c488cbaf6e0f0322b5c58 (table default.Table_2c04531a50c54791a64d6537b670d54f [id=e4025f53c2ea4b7aa8fafb057c102a3b]) successfully deleted
I20250629 01:51:49.236191 1158 ts_tablet_manager.cc:1905] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.239563 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:49.240229 1824 consensus_queue.cc:1035] T 955a41750b6a4766bb4fd3f01cfd99c8 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:49.240689 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 881461aa1bc34dd483f65e171c341bc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.242161 1154 tablet_bootstrap.cc:492] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.242262 1154 ts_tablet_manager.cc:1397] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20250629 01:51:49.242233 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2acf6246904d41ab94aad9dec7df2291" } modify_external_catalogs: true
I20250629 01:51:49.242393 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2acf6246904d41ab94aad9dec7df2291" } modify_external_catalogs: true
I20250629 01:51:49.242864 1154 raft_consensus.cc:357] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.242988 1154 raft_consensus.cc:383] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.243037 1154 raft_consensus.cc:738] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.243188 1154 consensus_queue.cc:260] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.243559 1154 ts_tablet_manager.cc:1428] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:49.243769 1154 tablet_bootstrap.cc:492] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.244033 1157 tablet_replica.cc:331] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:49.244258 1157 raft_consensus.cc:2241] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.244396 1157 raft_consensus.cc:2270] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.244884 1157 ts_tablet_manager.cc:1905] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.245044 1154 tablet_bootstrap.cc:654] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.250639 1155 tablet_bootstrap.cc:492] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.252755 1812 tablet.cc:1722] T 990d1fcdac1c406facb2da722421d7f4 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:49.253754 1158 ts_tablet_manager.cc:1918] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:49.253841 1158 log.cc:1199] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/82d58c88a6994d828b444c5814b287ff
I20250629 01:51:49.253913 1155 ts_tablet_manager.cc:1397] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.235s user 0.003s sys 0.000s
I20250629 01:51:49.254608 1154 tablet_bootstrap.cc:492] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.254714 1154 ts_tablet_manager.cc:1397] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20250629 01:51:49.255277 1154 raft_consensus.cc:357] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.255421 1154 raft_consensus.cc:383] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.255477 1154 raft_consensus.cc:738] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.255671 1154 consensus_queue.cc:260] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:49.253453 1818 negotiation.cc:337] Failed RPC negotiation. Trace:
0629 01:51:49.082410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:35873 (local address 127.0.104.2:56285)
0629 01:51:49.253143 (+170733us) negotiation.cc:107] Waiting for socket to connect
0629 01:51:49.253235 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":170645,"thread_start_us":170574,"threads_started":1}
I20250629 01:51:49.257014 1154 ts_tablet_manager.cc:1428] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:49.257928 1154 tablet_bootstrap.cc:492] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.259276 1154 tablet_bootstrap.cc:654] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.255383 1155 raft_consensus.cc:357] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:49.241240 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 28eaa55b3b564e2e9cf698fda98a4c8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28eaa55b3b564e2e9cf698fda98a4c8a already in progress: creating tablet
I20250629 01:51:49.260131 1155 raft_consensus.cc:383] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.260298 1155 raft_consensus.cc:738] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.260598 1155 consensus_queue.cc:260] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.263324 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_59e6d6754ff145b8bea3842035e15189" } modify_external_catalogs: true
I20250629 01:51:49.263494 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_59e6d6754ff145b8bea3842035e15189" } modify_external_catalogs: true
I20250629 01:51:49.263980 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99e79a9aa55444b6b6d1873d0d4844c8" } modify_external_catalogs: true
I20250629 01:51:49.264134 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99e79a9aa55444b6b6d1873d0d4844c8" } modify_external_catalogs: true
I20250629 01:51:49.267757 1801 raft_consensus.cc:491] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.267856 1801 raft_consensus.cc:513] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:49.260205 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3070ce8c467a47b0a833374450911291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3070ce8c467a47b0a833374450911291 already in progress: creating tablet
I20250629 01:51:49.268409 1801 leader_election.cc:290] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:49.268683 1159 tablet_bootstrap.cc:492] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:49.256541 1156 ts_tablet_manager.cc:1905] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.268779 1159 ts_tablet_manager.cc:1397] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.000s
I20250629 01:51:49.269033 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0ae8ed0950bd490598121af26c1d0eeb" } new_table_name: "default.Table_2657211e01984ab381578b3c21d851ea" modify_external_catalogs: true
W20250629 01:51:49.269101 850 leader_election.cc:343] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 5dfe921a84694903bbdb3fed3d6f2761
I20250629 01:51:49.269172 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_71248d2aeaee415194b2450bdb8def2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.269658 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71248d2aeaee415194b2450bdb8def2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:49.270324 850 leader_election.cc:343] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 5dfe921a84694903bbdb3fed3d6f2761
I20250629 01:51:49.270413 850 leader_election.cc:304] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.270587 1801 raft_consensus.cc:2747] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:49.271447 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0665d3f1225c44f894681fef4ae2520b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0665d3f1225c44f894681fef4ae2520b already in progress: creating tablet
W20250629 01:51:49.271786 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0827ec622ff04ca6b2fec50e40b9e1de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0827ec622ff04ca6b2fec50e40b9e1de already in progress: creating tablet
W20250629 01:51:49.271930 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f3bdc8bda8a940be865679a05b90bd08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3bdc8bda8a940be865679a05b90bd08 already in progress: creating tablet
W20250629 01:51:49.272082 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2249d5cba1544653b088312e7d74802e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2249d5cba1544653b088312e7d74802e already in progress: creating tablet
I20250629 01:51:49.273727 1154 tablet_bootstrap.cc:492] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.273811 1154 ts_tablet_manager.cc:1397] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250629 01:51:49.274406 1154 raft_consensus.cc:357] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.274519 1154 raft_consensus.cc:383] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.274557 1154 raft_consensus.cc:738] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.274677 1154 consensus_queue.cc:260] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.264477 1155 ts_tablet_manager.cc:1428] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250629 01:51:49.275274 1154 ts_tablet_manager.cc:1428] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:49.275480 1154 tablet_bootstrap.cc:492] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.276494 1155 tablet_bootstrap.cc:492] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.277051 1154 tablet_bootstrap.cc:654] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.277702 1155 tablet_bootstrap.cc:654] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.278596 1801 raft_consensus.cc:491] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.278688 1801 raft_consensus.cc:513] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:49.279351 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46c71d199da74227bd115c3a94178512 has no consensus state
I20250629 01:51:49.283571 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_57474f12b4c04c9c8a9e6e85838cdb8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:49.283671 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a20245e23a9e4dff9bfb1e565dc38184 (DEFAULT_TABLE table=default.Table_7ca2e46142c5424ab8f471c13d635ca2 [id=97190140b4a849aaa9758958f076c1db]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:49.284000 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57474f12b4c04c9c8a9e6e85838cdb8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.284190 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 46c71d199da74227bd115c3a94178512 (DEFAULT_TABLE table=default.Table_2657211e01984ab381578b3c21d851ea [id=7e04ac17360942d589ffe3b4158be18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.284510 747 tablet_service.cc:1468] Processing CreateTablet for tablet 46c71d199da74227bd115c3a94178512 (DEFAULT_TABLE table=default.Table_2657211e01984ab381578b3c21d851ea [id=7e04ac17360942d589ffe3b4158be18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.284780 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c71d199da74227bd115c3a94178512. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.288316 751 tablet_service.cc:1468] Processing CreateTablet for tablet e304f0bfe13846f29e6390d9904a4f57 (DEFAULT_TABLE table=default.Table_de40e2ec822b4c29a10c638c0a7d6f46 [id=896fee5246524ebabd2f166ae669fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.288600 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e304f0bfe13846f29e6390d9904a4f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.287809 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c71d199da74227bd115c3a94178512. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.294281 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c (DEFAULT_TABLE table=default.Table_2474fd6ed64c40278b389fe3aabc1765 [id=644e515f12b641e4b310e18faaa97ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.294587 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.294618 887 tablet_service.cc:1468] Processing CreateTablet for tablet a20245e23a9e4dff9bfb1e565dc38184 (DEFAULT_TABLE table=default.Table_7ca2e46142c5424ab8f471c13d635ca2 [id=97190140b4a849aaa9758958f076c1db]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:49.295329 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873): Couldn't send request to peer 0f4258c004e04bcf8216bb7c8fd56c54. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:49.295651 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5efd55e75b284a3f8c22890b9f24a104 (DEFAULT_TABLE table=default.Table_4afb0343bde845f3810f908c9ffb78ce [id=6b96debe16b1430095363fb872a5b6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.295919 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efd55e75b284a3f8c22890b9f24a104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.296314 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 5c83cc13930f4c99b9bc23603eef220f (DEFAULT_TABLE table=default.Table_bbfbc9cc8c0f441c8718c1ea68f3a287 [id=5df0399aa3b24f93aeffbed184416b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.296577 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c83cc13930f4c99b9bc23603eef220f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.299938 752 tablet_service.cc:1468] Processing CreateTablet for tablet 216c3d5318e0408db57371ea46db2930 (DEFAULT_TABLE table=default.Table_7e9bf1a6fab54b3198412022c8662ec0 [id=13a9d9c3e8fc48e0a3eb2423c0abb527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.300220 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 216c3d5318e0408db57371ea46db2930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.300287 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 19b642dd58e948329442964df736b4a4 (DEFAULT_TABLE table=default.Table_1c1f88cfa7a84329b2ffe152996d178e [id=63f3f9592ddf4120916a1a602ee3d605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.300566 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b642dd58e948329442964df736b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.304284 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 2540a84a922949e3adb90d0cded1c951 (DEFAULT_TABLE table=default.Table_d499c5244b63442cbb0110787b2c209d [id=b88a2362b5114d78815e8eec1148a322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.304548 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540a84a922949e3adb90d0cded1c951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.306444 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a32636c293bc44858da7f29a515c467b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.306708 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0e795729238f49c29b88fa9639d82976 (DEFAULT_TABLE table=default.Table_d86465b0bdc64cf384232c87c1b67862 [id=87d1c516338f405b8a306c23f4eb8e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.306927 740 tablet_service.cc:1468] Processing CreateTablet for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 (DEFAULT_TABLE table=default.Table_709271d3d173470ab9aace403ba9d5e3 [id=66634c8f270c44a8ad903b7f6b0f8321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.307413 739 tablet_service.cc:1468] Processing CreateTablet for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b (DEFAULT_TABLE table=default.Table_218288faefcf4b5d9df5adbf5d4223cb [id=d5be43fadc9a47909da94c26429e6589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.307480 746 tablet_service.cc:1468] Processing CreateTablet for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d (DEFAULT_TABLE table=default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.307839 750 tablet_service.cc:1468] Processing CreateTablet for tablet 5c83cc13930f4c99b9bc23603eef220f (DEFAULT_TABLE table=default.Table_bbfbc9cc8c0f441c8718c1ea68f3a287 [id=5df0399aa3b24f93aeffbed184416b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.307891 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4234c00081e847e9ba6bb70e630ba764 (DEFAULT_TABLE table=default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 [id=2e90d163f6834cf79730e835b629994a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.308270 749 tablet_service.cc:1468] Processing CreateTablet for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c (DEFAULT_TABLE table=default.Table_2474fd6ed64c40278b389fe3aabc1765 [id=644e515f12b641e4b310e18faaa97ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.308357 738 tablet_service.cc:1468] Processing CreateTablet for tablet 0bef2747f0c04f498079906e40bac24c (DEFAULT_TABLE table=default.Table_99e79a9aa55444b6b6d1873d0d4844c8 [id=c483797822d647458b5728bf3255b15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.308645 741 tablet_service.cc:1468] Processing CreateTablet for tablet 2540a84a922949e3adb90d0cded1c951 (DEFAULT_TABLE table=default.Table_d499c5244b63442cbb0110787b2c209d [id=b88a2362b5114d78815e8eec1148a322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.309051 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 0bef2747f0c04f498079906e40bac24c (DEFAULT_TABLE table=default.Table_99e79a9aa55444b6b6d1873d0d4844c8 [id=c483797822d647458b5728bf3255b15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.309065 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_296cbd48cdc743689498e69b6d8c1a7c" } new_table_name: "default.Table_23573af00cf74adb8eee3267f019a3ee" modify_external_catalogs: true
I20250629 01:51:49.309330 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bef2747f0c04f498079906e40bac24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.309543 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 4234c00081e847e9ba6bb70e630ba764 (DEFAULT_TABLE table=default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 [id=2e90d163f6834cf79730e835b629994a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.309794 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4234c00081e847e9ba6bb70e630ba764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.312222 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.312407 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0e795729238f49c29b88fa9639d82976 (DEFAULT_TABLE table=default.Table_d86465b0bdc64cf384232c87c1b67862 [id=87d1c516338f405b8a306c23f4eb8e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.312513 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d (DEFAULT_TABLE table=default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.312704 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e795729238f49c29b88fa9639d82976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.312886 1023 tablet_service.cc:1468] Processing CreateTablet for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 (DEFAULT_TABLE table=default.Table_709271d3d173470ab9aace403ba9d5e3 [id=66634c8f270c44a8ad903b7f6b0f8321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.313212 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.315955 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet e82b1352bb4d49769808dc402c85890f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.316165 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f0ae4c51924370896c624affde6761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.309335 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b (DEFAULT_TABLE table=default.Table_218288faefcf4b5d9df5adbf5d4223cb [id=d5be43fadc9a47909da94c26429e6589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.316757 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.317098 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5d2f2975e5d64efd888ed948f1420916 (DEFAULT_TABLE table=default.Table_3699a1e4f7ec403ba943711d0c3e6588 [id=cd3d373467b54508a8edfc36da09c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.317425 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2f2975e5d64efd888ed948f1420916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.320276 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.320434 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5d2f2975e5d64efd888ed948f1420916 (DEFAULT_TABLE table=default.Table_3699a1e4f7ec403ba943711d0c3e6588 [id=cd3d373467b54508a8edfc36da09c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.320855 754 tablet_service.cc:1468] Processing CreateTablet for tablet 19b642dd58e948329442964df736b4a4 (DEFAULT_TABLE table=default.Table_1c1f88cfa7a84329b2ffe152996d178e [id=63f3f9592ddf4120916a1a602ee3d605]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.324748 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b15fad4b4b3c403daa112af034e6e3d8" } new_table_name: "default.Table_8715038ee85a4103b95aea0f5f85b2e6" modify_external_catalogs: true
I20250629 01:51:49.306989 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e795729238f49c29b88fa9639d82976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.326145 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2f2975e5d64efd888ed948f1420916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.326300 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.326476 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540a84a922949e3adb90d0cded1c951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.329123 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.329627 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4234c00081e847e9ba6bb70e630ba764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.330047 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.332012 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b642dd58e948329442964df736b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.332160 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.334540 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c83cc13930f4c99b9bc23603eef220f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.335958 1710 raft_consensus.cc:491] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.336107 1710 raft_consensus.cc:513] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.336601 1710 leader_election.cc:290] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:49.337034 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5150eff1ad43588a9e70dab47744e3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:49.337498 986 leader_election.cc:343] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.338099 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bef2747f0c04f498079906e40bac24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.292668 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cbd53c3618ee416a842d119b503fa368"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.343276 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbd53c3618ee416a842d119b503fa368 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.344584 885 tablet_service.cc:1468] Processing CreateTablet for tablet 46c71d199da74227bd115c3a94178512 (DEFAULT_TABLE table=default.Table_2657211e01984ab381578b3c21d851ea [id=7e04ac17360942d589ffe3b4158be18f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.344668 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 216c3d5318e0408db57371ea46db2930 (DEFAULT_TABLE table=default.Table_7e9bf1a6fab54b3198412022c8662ec0 [id=13a9d9c3e8fc48e0a3eb2423c0abb527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.344949 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 216c3d5318e0408db57371ea46db2930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.345559 757 tablet_service.cc:1468] Processing CreateTablet for tablet a20245e23a9e4dff9bfb1e565dc38184 (DEFAULT_TABLE table=default.Table_7ca2e46142c5424ab8f471c13d635ca2 [id=97190140b4a849aaa9758958f076c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.345903 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20245e23a9e4dff9bfb1e565dc38184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.349526 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0bef2747f0c04f498079906e40bac24c
I20250629 01:51:49.349807 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 5efd55e75b284a3f8c22890b9f24a104 (DEFAULT_TABLE table=default.Table_4afb0343bde845f3810f908c9ffb78ce [id=6b96debe16b1430095363fb872a5b6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.350164 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efd55e75b284a3f8c22890b9f24a104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.353896 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20245e23a9e4dff9bfb1e565dc38184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.356750 1158 ts_tablet_manager.cc:1939] T 82d58c88a6994d828b444c5814b287ff P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:49.357234 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bef2747f0c04f498079906e40bac24c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.359171 1032 tablet_service.cc:1468] Processing CreateTablet for tablet e304f0bfe13846f29e6390d9904a4f57 (DEFAULT_TABLE table=default.Table_de40e2ec822b4c29a10c638c0a7d6f46 [id=896fee5246524ebabd2f166ae669fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.359272 1158 tablet_replica.cc:331] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:49.359541 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e304f0bfe13846f29e6390d9904a4f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.368302 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]) successfully deleted
I20250629 01:51:49.370563 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bef2747f0c04f498079906e40bac24c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.375149 1048 raft_consensus.cc:1273] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:49.375598 1766 consensus_queue.cc:1035] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:49.383360 1156 ts_tablet_manager.cc:1918] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.383599 1801 leader_election.cc:290] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:49.384348 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0" } modify_external_catalogs: true
I20250629 01:51:49.384435 881 tablet_service.cc:1468] Processing CreateTablet for tablet e304f0bfe13846f29e6390d9904a4f57 (DEFAULT_TABLE table=default.Table_de40e2ec822b4c29a10c638c0a7d6f46 [id=896fee5246524ebabd2f166ae669fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.384514 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3c62eff24dc14bc9ad9091f4a06ef8a0" } modify_external_catalogs: true
I20250629 01:51:49.384826 893 tablet_service.cc:1468] Processing CreateTablet for tablet 216c3d5318e0408db57371ea46db2930 (DEFAULT_TABLE table=default.Table_7e9bf1a6fab54b3198412022c8662ec0 [id=13a9d9c3e8fc48e0a3eb2423c0abb527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.384975 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d96fb8679e6648e0af1e38be2e2bc67f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.385404 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d96fb8679e6648e0af1e38be2e2bc67f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.384363 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f9284d7ccab1429d91910af2fd53b89c" } new_table_name: "default.Table_c58b64be9b9040a294ce611a1bd282c7" modify_external_catalogs: true
I20250629 01:51:49.388453 877 tablet_service.cc:1468] Processing CreateTablet for tablet 5efd55e75b284a3f8c22890b9f24a104 (DEFAULT_TABLE table=default.Table_4afb0343bde845f3810f908c9ffb78ce [id=6b96debe16b1430095363fb872a5b6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.389010 880 tablet_service.cc:1468] Processing CreateTablet for tablet 5c83cc13930f4c99b9bc23603eef220f (DEFAULT_TABLE table=default.Table_bbfbc9cc8c0f441c8718c1ea68f3a287 [id=5df0399aa3b24f93aeffbed184416b43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.392287 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd5150eff1ad43588a9e70dab47744e3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:49.392791 915 raft_consensus.cc:3058] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:49.392879 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0bef2747f0c04f498079906e40bac24c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.359547 1158 raft_consensus.cc:2241] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.393046 1158 raft_consensus.cc:2270] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.393491 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_df8a368b4df84492add9218c8d1deb22" } modify_external_catalogs: true
I20250629 01:51:49.393608 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_df8a368b4df84492add9218c8d1deb22" } modify_external_catalogs: true
I20250629 01:51:49.394615 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7054a2ae6968462bbf1a604dc08243fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.395011 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7054a2ae6968462bbf1a604dc08243fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.396027 895 tablet_service.cc:1468] Processing CreateTablet for tablet 2540a84a922949e3adb90d0cded1c951 (DEFAULT_TABLE table=default.Table_d499c5244b63442cbb0110787b2c209d [id=b88a2362b5114d78815e8eec1148a322]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:49.397163 850 leader_election.cc:343] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: d663e7dc7a384bf6bd0cbfa9d9e5e9f0
W20250629 01:51:49.397300 850 leader_election.cc:343] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d663e7dc7a384bf6bd0cbfa9d9e5e9f0
I20250629 01:51:49.397351 850 leader_election.cc:304] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.398416 1801 raft_consensus.cc:2747] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:49.398432 988 leader_election.cc:343] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.398520 988 leader_election.cc:304] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.398805 1710 raft_consensus.cc:2747] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.399904 1804 tablet.cc:1722] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:49.395627 1808 tablet.cc:1722] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:49.400481 1156 log.cc:1199] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d61687e19d994e87b09e55bd3b9476d7
I20250629 01:51:49.400758 1156 ts_tablet_manager.cc:1939] T d61687e19d994e87b09e55bd3b9476d7 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.401470 1156 tablet_replica.cc:331] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:49.401618 1156 raft_consensus.cc:2241] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.401607 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d61687e19d994e87b09e55bd3b9476d7 (table default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb [id=f274d40e73944253825a87b39ee56b71]) successfully deleted
I20250629 01:51:49.401723 1156 raft_consensus.cc:2270] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.402175 886 tablet_service.cc:1468] Processing CreateTablet for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c (DEFAULT_TABLE table=default.Table_2474fd6ed64c40278b389fe3aabc1765 [id=644e515f12b641e4b310e18faaa97ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.402307 884 tablet_service.cc:1468] Processing CreateTablet for tablet 5d2f2975e5d64efd888ed948f1420916 (DEFAULT_TABLE table=default.Table_3699a1e4f7ec403ba943711d0c3e6588 [id=cd3d373467b54508a8edfc36da09c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.402871 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 22567416bfde4e2b843cf5c7a214ec6c
I20250629 01:51:49.405512 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 22567416bfde4e2b843cf5c7a214ec6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.405987 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 22567416bfde4e2b843cf5c7a214ec6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.406517 1156 ts_tablet_manager.cc:1905] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.406608 455 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:49.408025 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]). Expected version 2 got 0
I20250629 01:51:49.408123 1158 ts_tablet_manager.cc:1905] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.408699 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c525edb71817495cbedf4ec757b7810e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.409107 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c525edb71817495cbedf4ec757b7810e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.412778 1155 tablet_bootstrap.cc:492] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.412879 1155 ts_tablet_manager.cc:1397] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.136s user 0.001s sys 0.001s
I20250629 01:51:49.412815 1157 ts_tablet_manager.cc:1918] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.413168 890 tablet_service.cc:1468] Processing CreateTablet for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d (DEFAULT_TABLE table=default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.413321 878 tablet_service.cc:1468] Processing CreateTablet for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b (DEFAULT_TABLE table=default.Table_218288faefcf4b5d9df5adbf5d4223cb [id=d5be43fadc9a47909da94c26429e6589]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.413471 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 22567416bfde4e2b843cf5c7a214ec6c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.413429 1155 raft_consensus.cc:357] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.413554 1155 raft_consensus.cc:383] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.413589 1155 raft_consensus.cc:738] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.413569 882 tablet_service.cc:1468] Processing CreateTablet for tablet 4234c00081e847e9ba6bb70e630ba764 (DEFAULT_TABLE table=default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851 [id=2e90d163f6834cf79730e835b629994a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.413727 1155 consensus_queue.cc:260] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.414592 888 tablet_service.cc:1468] Processing CreateTablet for tablet 0e795729238f49c29b88fa9639d82976 (DEFAULT_TABLE table=default.Table_d86465b0bdc64cf384232c87c1b67862 [id=87d1c516338f405b8a306c23f4eb8e4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.414758 896 tablet_service.cc:1468] Processing CreateTablet for tablet 0bef2747f0c04f498079906e40bac24c (DEFAULT_TABLE table=default.Table_99e79a9aa55444b6b6d1873d0d4844c8 [id=c483797822d647458b5728bf3255b15d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.414860 889 tablet_service.cc:1468] Processing CreateTablet for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 (DEFAULT_TABLE table=default.Table_709271d3d173470ab9aace403ba9d5e3 [id=66634c8f270c44a8ad903b7f6b0f8321]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.414994 1155 ts_tablet_manager.cc:1428] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:49.415202 1155 tablet_bootstrap.cc:492] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.416553 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b
I20250629 01:51:49.417022 879 tablet_service.cc:1468] Processing CreateTablet for tablet 19b642dd58e948329442964df736b4a4 (DEFAULT_TABLE table=default.Table_1c1f88cfa7a84329b2ffe152996d178e [id=63f3f9592ddf4120916a1a602ee3d605]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:49.418439 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b0ef1ba46ac487ab1502c3d116814e1 consensus state has no leader
I20250629 01:51:49.421958 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:49.423034 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbe1f8083a884cdd97afce83bd22073c consensus state has no leader
W20250629 01:51:49.423301 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd9a41fae477470281dfba4d4e552b99 consensus state has no leader
I20250629 01:51:49.423398 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4aa8dfd712c043ad9f8ff56b064bb8fd
I20250629 01:51:49.423622 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a254b0317477416389b93521d375802b" } new_table_name: "default.Table_d61ce7701ae64cfd903fdcdbc989bc74" modify_external_catalogs: true
I20250629 01:51:49.423913 1159 raft_consensus.cc:357] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.423957 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dc942d4ae58a4c82a26b62777b048719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.424379 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc942d4ae58a4c82a26b62777b048719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.424863 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6217ce5beab343c0af71e61d491db6f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:49.425040 1159 raft_consensus.cc:383] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.425133 1159 raft_consensus.cc:738] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:51:49.425261 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6217ce5beab343c0af71e61d491db6f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.423055 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_660af79d2b29409fae344ba080c4519a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.425854 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_660af79d2b29409fae344ba080c4519a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.427318 1159 consensus_queue.cc:260] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.427850 1159 ts_tablet_manager.cc:1428] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.159s user 0.000s sys 0.002s
I20250629 01:51:49.430593 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7c434928fb344af2aebbe8eb3f5376a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.430963 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c434928fb344af2aebbe8eb3f5376a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.431756 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:49.438474 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9dad47c452ca45039f44d08c43a7cca3 consensus state has no leader
I20250629 01:51:49.438500 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c] alter complete (version 2)
I20250629 01:51:49.439049 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.442983 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.443607 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.444285 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.446660 1155 tablet_bootstrap.cc:654] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.452054 1824 raft_consensus.cc:491] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.452176 1824 raft_consensus.cc:513] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.452630 1824 leader_election.cc:290] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:49.456822 710 leader_election.cc:343] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a32636c293bc44858da7f29a515c467b
I20250629 01:51:49.464792 1155 tablet_bootstrap.cc:492] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.464926 1155 ts_tablet_manager.cc:1397] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.002s
I20250629 01:51:49.465485 1155 raft_consensus.cc:357] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.465668 1155 raft_consensus.cc:383] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.465760 1155 raft_consensus.cc:738] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.465896 1156 ts_tablet_manager.cc:1918] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.471855 1158 ts_tablet_manager.cc:1918] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.465986 1155 consensus_queue.cc:260] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.472465 1155 ts_tablet_manager.cc:1428] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250629 01:51:49.472677 1155 tablet_bootstrap.cc:492] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.473974 1155 tablet_bootstrap.cc:654] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.486300 1155 tablet_bootstrap.cc:492] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.486461 1155 ts_tablet_manager.cc:1397] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20250629 01:51:49.487036 1155 raft_consensus.cc:357] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.487221 1155 raft_consensus.cc:383] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.487291 1155 raft_consensus.cc:738] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.487609 1155 consensus_queue.cc:260] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.490012 1155 ts_tablet_manager.cc:1428] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:49.490411 1155 tablet_bootstrap.cc:492] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.491885 1155 tablet_bootstrap.cc:654] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.492650 1156 log.cc:1199] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/75fd4ffcf896419b81a1c0eb7c5aeeec
I20250629 01:51:49.492959 1156 ts_tablet_manager.cc:1939] T 75fd4ffcf896419b81a1c0eb7c5aeeec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.493884 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 75fd4ffcf896419b81a1c0eb7c5aeeec (table default.Table_5ce4c958f3d4456fb56450ca4cc26221 [id=646791dd9bb447fdb2e96a9046dd68d2]) successfully deleted
I20250629 01:51:49.494345 1156 tablet_replica.cc:331] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:49.494549 1156 raft_consensus.cc:2241] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.494704 1156 raft_consensus.cc:2270] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:49.494751 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 17acb9bcd7aa4a508d1695474a1b56df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17acb9bcd7aa4a508d1695474a1b56df already in progress: creating tablet
W20250629 01:51:49.495015 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 52be184d0480495a894442d6f72a0f12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52be184d0480495a894442d6f72a0f12 already in progress: creating tablet
W20250629 01:51:49.495193 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 77eae224763d464496066b4a98efe5ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77eae224763d464496066b4a98efe5ff already in progress: creating tablet
I20250629 01:51:49.495288 1156 ts_tablet_manager.cc:1905] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:49.495469 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ca13eb6bdcd044ec948ea699857a30f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca13eb6bdcd044ec948ea699857a30f8 already in progress: creating tablet
W20250629 01:51:49.495637 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bdc6a26b3594d6fb33a27cac5b29f4f already in progress: creating tablet
W20250629 01:51:49.495842 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f94d8fc1b6254ba69acadcaff3613d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f94d8fc1b6254ba69acadcaff3613d97 already in progress: creating tablet
I20250629 01:51:49.497382 1154 tablet_bootstrap.cc:492] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.497844 1158 log.cc:1199] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/86832a3c26224c7ea0b9b4b6866d3880
I20250629 01:51:49.497931 1154 ts_tablet_manager.cc:1397] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.223s user 0.022s sys 0.077s
I20250629 01:51:49.498653 1158 ts_tablet_manager.cc:1939] T 86832a3c26224c7ea0b9b4b6866d3880 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:49.498849 1154 raft_consensus.cc:357] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.499958 1154 raft_consensus.cc:383] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.500052 1154 raft_consensus.cc:738] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.500313 1154 consensus_queue.cc:260] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.500787 1158 tablet_replica.cc:331] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:49.500924 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 86832a3c26224c7ea0b9b4b6866d3880 (table default.Table_f0a26d9ea3e8407da5ca0813db26beb3 [id=b52a4204ccf34819b3782aca3ea1db0f]) successfully deleted
I20250629 01:51:49.500985 1158 raft_consensus.cc:2241] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.501088 1158 raft_consensus.cc:2270] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:49.501159 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 17acb9bcd7aa4a508d1695474a1b56df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17acb9bcd7aa4a508d1695474a1b56df already in progress: creating tablet
W20250629 01:51:49.501346 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 52be184d0480495a894442d6f72a0f12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52be184d0480495a894442d6f72a0f12 already in progress: creating tablet
I20250629 01:51:49.503084 1154 ts_tablet_manager.cc:1428] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:49.503312 1155 tablet_bootstrap.cc:492] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.503451 1155 ts_tablet_manager.cc:1397] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20250629 01:51:49.503610 1158 ts_tablet_manager.cc:1905] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.504004 1155 raft_consensus.cc:357] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.504159 1155 raft_consensus.cc:383] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.504205 1155 raft_consensus.cc:738] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.504355 1154 tablet_bootstrap.cc:492] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.504344 1155 consensus_queue.cc:260] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.504689 1155 ts_tablet_manager.cc:1428] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:49.504899 1155 tablet_bootstrap.cc:492] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.505717 1154 tablet_bootstrap.cc:654] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.506074 1155 tablet_bootstrap.cc:654] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.511273 1155 tablet_bootstrap.cc:492] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:49.511426 1155 ts_tablet_manager.cc:1397] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250629 01:51:49.512188 1155 raft_consensus.cc:357] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.512382 1155 raft_consensus.cc:383] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.512470 1155 raft_consensus.cc:738] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.512665 1155 consensus_queue.cc:260] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.513046 1155 ts_tablet_manager.cc:1428] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:49.513289 1155 tablet_bootstrap.cc:492] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:49.514720 1155 tablet_bootstrap.cc:654] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.515357 1154 tablet_bootstrap.cc:492] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:49.515503 1154 ts_tablet_manager.cc:1397] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20250629 01:51:49.516144 1154 raft_consensus.cc:357] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.516325 1154 raft_consensus.cc:383] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.516414 1154 raft_consensus.cc:738] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.516599 1154 consensus_queue.cc:260] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.517778 1154 ts_tablet_manager.cc:1428] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:49.518766 1154 tablet_bootstrap.cc:492] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:49.520030 1154 tablet_bootstrap.cc:654] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.521724 1158 ts_tablet_manager.cc:1918] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.527372 1156 ts_tablet_manager.cc:1918] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.527455 1156 log.cc:1199] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/28eaa55b3b564e2e9cf698fda98a4c8a
I20250629 01:51:49.527690 1156 ts_tablet_manager.cc:1939] T 28eaa55b3b564e2e9cf698fda98a4c8a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.528368 1156 tablet_replica.cc:331] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:49.528515 1156 raft_consensus.cc:2241] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.528596 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 28eaa55b3b564e2e9cf698fda98a4c8a (table default.Table_8b976eb77218429b9513d370b45cd5c9 [id=67701e562ee84eaf9030d915a726ebe2]) successfully deleted
I20250629 01:51:49.530078 1158 log.cc:1199] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/bd5150eff1ad43588a9e70dab47744e3
I20250629 01:51:49.530339 1158 ts_tablet_manager.cc:1939] T bd5150eff1ad43588a9e70dab47744e3 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:49.530956 1158 tablet_replica.cc:331] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:49.531083 1158 raft_consensus.cc:2241] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.531186 1158 raft_consensus.cc:2270] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.528626 1156 raft_consensus.cc:2270] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.534827 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a20245e23a9e4dff9bfb1e565dc38184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.536343 1159 tablet_bootstrap.cc:492] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:49.536765 1156 ts_tablet_manager.cc:1905] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.538286 1158 ts_tablet_manager.cc:1905] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.546370 1156 ts_tablet_manager.cc:1918] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.556617 1156 log.cc:1199] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a32636c293bc44858da7f29a515c467b
I20250629 01:51:49.556989 1156 ts_tablet_manager.cc:1939] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.557654 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a32636c293bc44858da7f29a515c467b (table default.Table_2fc84dd48d464ce682425fbd41d92b7e [id=64b627ee531b41ffa46d4bbd813c5b8c]) successfully deleted
W20250629 01:51:49.558339 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0bef2747f0c04f498079906e40bac24c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bef2747f0c04f498079906e40bac24c already in progress: creating tablet
W20250629 01:51:49.558986 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 22567416bfde4e2b843cf5c7a214ec6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22567416bfde4e2b843cf5c7a214ec6c already in progress: creating tablet
W20250629 01:51:49.559607 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b already in progress: creating tablet
W20250629 01:51:49.560235 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa8dfd712c043ad9f8ff56b064bb8fd already in progress: creating tablet
I20250629 01:51:49.567058 1157 log.cc:1199] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8bcc3f2a76bc4bae8b306b54e75656cc
I20250629 01:51:49.567190 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a32636c293bc44858da7f29a515c467b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:49.567116 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.572463 712 leader_election.cc:343] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.572641 712 leader_election.cc:304] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:49.572865 712 raft_consensus.cc:2684] T a32636c293bc44858da7f29a515c467b P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:49.579223 1157 ts_tablet_manager.cc:1939] T 8bcc3f2a76bc4bae8b306b54e75656cc P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:49.579694 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4234c00081e847e9ba6bb70e630ba764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.581907 1159 tablet_bootstrap.cc:654] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.594282 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.596235 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e795729238f49c29b88fa9639d82976. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.599942 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e304f0bfe13846f29e6390d9904a4f57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.607694 1710 raft_consensus.cc:491] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.607833 1710 raft_consensus.cc:513] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.608520 1710 leader_election.cc:290] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:49.610108 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0827ec622ff04ca6b2fec50e40b9e1de" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:49.610778 986 leader_election.cc:343] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.621407 915 raft_consensus.cc:1273] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:49.622094 1824 consensus_queue.cc:1035] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:49.633109 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0827ec622ff04ca6b2fec50e40b9e1de" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:49.633422 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 216c3d5318e0408db57371ea46db2930. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.640447 988 leader_election.cc:343] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.640563 988 leader_election.cc:304] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.640983 1710 raft_consensus.cc:2747] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.644275 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c71d199da74227bd115c3a94178512. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.644493 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bef2747f0c04f498079906e40bac24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.644841 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c83cc13930f4c99b9bc23603eef220f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.645174 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19b642dd58e948329442964df736b4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.645962 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2540a84a922949e3adb90d0cded1c951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.647652 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d2f2975e5d64efd888ed948f1420916. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.650475 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 627) took 1189 ms. Trace:
I20250629 01:51:49.650624 1157 rpcz_store.cc:276] 0629 01:51:48.454714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.499982 (+ 45268us) service_pool.cc:225] Handling call
0629 01:51:49.243988 (+744006us) ts_tablet_manager.cc:1145] Deleting tablet 8bcc3f2a76bc4bae8b306b54e75656cc
0629 01:51:49.243997 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.385099 (+141102us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:49.412721 (+ 27622us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:49.644167 (+231446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":738045}
I20250629 01:51:49.651212 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8bcc3f2a76bc4bae8b306b54e75656cc (table default.Table_f6f27d5499af49359886879547b0ac31 [id=e1d26b8e31044413af1d4bddc58816e7]) successfully deleted
I20250629 01:51:49.651655 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 630) took 1039 ms. Trace:
I20250629 01:51:49.651773 1157 rpcz_store.cc:276] 0629 01:51:48.612356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.612507 (+ 151us) service_pool.cc:225] Handling call
0629 01:51:49.651541 (+1039034us) ts_tablet_manager.cc:1145] Deleting tablet 897a4cefefe04f06b47070ec7fa1a33b
0629 01:51:49.651552 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.651640 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038843}
W20250629 01:51:49.652283 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 897a4cefefe04f06b47070ec7fa1a33b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897a4cefefe04f06b47070ec7fa1a33b already in progress: creating tablet
I20250629 01:51:49.652690 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 631) took 1040 ms. Trace:
I20250629 01:51:49.652801 1157 rpcz_store.cc:276] 0629 01:51:48.612467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.612717 (+ 250us) service_pool.cc:225] Handling call
0629 01:51:49.652585 (+1039868us) ts_tablet_manager.cc:1145] Deleting tablet 4d9ef77181ba48f3a6de16b945c8df1a
0629 01:51:49.652597 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.652674 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039736}
W20250629 01:51:49.653264 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4d9ef77181ba48f3a6de16b945c8df1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d9ef77181ba48f3a6de16b945c8df1a already in progress: creating tablet
W20250629 01:51:49.653872 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b75d33594c994363bdb2558eb0c6ca3d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b75d33594c994363bdb2558eb0c6ca3d already in progress: creating tablet
W20250629 01:51:49.654472 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 258994dd24804f4aaa5e299436fac9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 258994dd24804f4aaa5e299436fac9b0 already in progress: creating tablet
W20250629 01:51:49.655036 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 871086e50583453cadcc60ccd8afcba6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 871086e50583453cadcc60ccd8afcba6 already in progress: creating tablet
I20250629 01:51:49.660651 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.661715 1157 tablet_replica.cc:331] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:49.661957 1157 raft_consensus.cc:2241] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.662358 1710 raft_consensus.cc:491] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.662501 1710 raft_consensus.cc:513] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.663089 1710 leader_election.cc:290] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:49.664613 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e82b1352bb4d49769808dc402c85890f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:49.665051 986 leader_election.cc:343] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.663862 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e82b1352bb4d49769808dc402c85890f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:49.676937 988 leader_election.cc:343] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.677042 988 leader_election.cc:304] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.677438 1710 raft_consensus.cc:2747] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.677752 1157 raft_consensus.cc:2270] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.678324 1157 ts_tablet_manager.cc:1905] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.678696 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efd55e75b284a3f8c22890b9f24a104. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.681053 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.730531 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 63595.8us
I20250629 01:51:49.757292 1812 tablet.cc:1722] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:49.775329 1836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:49.775653 1836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:51:49.775964 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_39cde32ddcd642cc8b9240f4d3f979a6" } new_table_name: "default.Table_581eff9507d54eb092e4ad71badc3c08" modify_external_catalogs: true
I20250629 01:51:49.776684 1159 tablet_bootstrap.cc:492] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:49.776803 1159 ts_tablet_manager.cc:1397] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.319s user 0.000s sys 0.006s
I20250629 01:51:49.777199 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7f95c64f736b4aee86abc0c7785ab8d6" } modify_external_catalogs: true
I20250629 01:51:49.777357 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7f95c64f736b4aee86abc0c7785ab8d6" } modify_external_catalogs: true
I20250629 01:51:49.777539 1159 raft_consensus.cc:357] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.777670 1159 raft_consensus.cc:383] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:49.777712 1159 raft_consensus.cc:738] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:49.777772 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d20078e8bc4e4a218577b8bc71fc8d44" } modify_external_catalogs: true
I20250629 01:51:49.777896 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d20078e8bc4e4a218577b8bc71fc8d44" } modify_external_catalogs: true
I20250629 01:51:49.777962 1159 consensus_queue.cc:260] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.778337 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_158439966f9d48839815a9c82d78c1ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.778777 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_158439966f9d48839815a9c82d78c1ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.775964 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_85166d71c7d74a25be9d12b625da1f5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:49.781760 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9" } modify_external_catalogs: true
I20250629 01:51:49.781893 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9" } modify_external_catalogs: true
W20250629 01:51:49.782357 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85166d71c7d74a25be9d12b625da1f5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.782418 1159 ts_tablet_manager.cc:1428] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250629 01:51:49.783315 1159 tablet_bootstrap.cc:492] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:49.784767 1159 tablet_bootstrap.cc:654] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:49.791801 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a403c9cdeab14d55a3defb5ae2056216" } new_table_name: "default.Table_992098655a084dd1a8d522be5b7c1cc3" modify_external_catalogs: true
I20250629 01:51:49.792907 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 96324ddf32a1460b8fc37e2c324649e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 0827ec622ff04ca6b2fec50e40b9e1de) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.798105 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 26d85fd3d3dd4b07a812c019e1a5c748 (table default.Table_331d3b611f4f46748132e37fb1a8e872 [id=b9df439e13104b14be7c1b522b04ec13]). Expected version 4 got 0
I20250629 01:51:49.798753 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_966c88ce78e148fe9366b092be734d02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.799023 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3874404c3444d47a71f1dce7fb86ff8 consensus state has no leader
W20250629 01:51:49.799139 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_966c88ce78e148fe9366b092be734d02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.799328 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b172df6df0ab4b738c77a9835b3f88c2" } new_table_name: "default.Table_8a62929d23e2422e979440dd100634a5" modify_external_catalogs: true
I20250629 01:51:49.801636 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_30b8e1c0bc8241a7ba265c3c4f86a095" } modify_external_catalogs: true
I20250629 01:51:49.801713 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7edfdf771ecd4ae8b87791846536271f (DEFAULT_TABLE table=default.Table_0c86a89fdf674dfc856f5a107daad436 [id=af0787145f544902be2d758b2b957d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.801777 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_30b8e1c0bc8241a7ba265c3c4f86a095" } modify_external_catalogs: true
I20250629 01:51:49.802073 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edfdf771ecd4ae8b87791846536271f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.802819 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:49.803633 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7f38778518cf47bca0a3dc0dc8e94313" } new_table_name: "default.Table_a2726ab5a15a4cf8833868289c7ca5ee" modify_external_catalogs: true
I20250629 01:51:49.803881 754 tablet_service.cc:1468] Processing CreateTablet for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 (DEFAULT_TABLE table=default.Table_415759429f02467f944d3c1a215d5966 [id=890bc27877e4475d8a573fa60c08055e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.804293 740 tablet_service.cc:1468] Processing CreateTablet for tablet d02d437359a643b59d1bd41dfe77a0fc (DEFAULT_TABLE table=default.Table_71248d2aeaee415194b2450bdb8def2e [id=f6381e088b094625be789e5010713590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.804468 745 tablet_service.cc:1468] Processing CreateTablet for tablet ce7a72db953646038a9d82dde04d76e4 (DEFAULT_TABLE table=default.Table_3ec720103be44b36a6f7d15307a437ff [id=12467795994e4dbaaa74e37c8b9cef32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.804718 750 tablet_service.cc:1468] Processing CreateTablet for tablet bbfcc7513562418fa634b873beac2c4f (DEFAULT_TABLE table=default.Table_61af8dc03a3049d59036eab5d6240298 [id=ae30602e16274e1eb86d0a7013d8f249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.804293 743 tablet_service.cc:1468] Processing CreateTablet for tablet 2776d8b893c94c099b63bb93030ca714 (DEFAULT_TABLE table=default.Table_7bb26e59c27447b09a8cacbad6133b3a [id=0aadedc9390a4a28ad6c9db7c527eded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.805250 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.805476 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b1aaca780fe54c5dbbba0d76d7194624"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:49.805627 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 69db100a14554271aee32c3f11af4346 (DEFAULT_TABLE table=default.Table_34b52be75f5d4681945cc00f5911c2d2 [id=562105e13fb64deebf82e76813e63002]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:49.805809 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1aaca780fe54c5dbbba0d76d7194624 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.805909 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69db100a14554271aee32c3f11af4346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.807723 746 tablet_service.cc:1468] Processing CreateTablet for tablet 69db100a14554271aee32c3f11af4346 (DEFAULT_TABLE table=default.Table_34b52be75f5d4681945cc00f5911c2d2 [id=562105e13fb64deebf82e76813e63002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.808076 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69db100a14554271aee32c3f11af4346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.808295 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 2776d8b893c94c099b63bb93030ca714 (DEFAULT_TABLE table=default.Table_7bb26e59c27447b09a8cacbad6133b3a [id=0aadedc9390a4a28ad6c9db7c527eded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.808560 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2776d8b893c94c099b63bb93030ca714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.810523 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02d437359a643b59d1bd41dfe77a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.811739 738 tablet_service.cc:1468] Processing CreateTablet for tablet 9c90ec6864664fe48d8b9a92837b748c (DEFAULT_TABLE table=default.Table_3dd8aa3c2e914f019c0e7d29a3dbda13 [id=f55e6319c9174c39b0a8371818509111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.811988 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c90ec6864664fe48d8b9a92837b748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.812192 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbfcc7513562418fa634b873beac2c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.812299 1015 tablet_service.cc:1468] Processing CreateTablet for tablet ce7a72db953646038a9d82dde04d76e4 (DEFAULT_TABLE table=default.Table_3ec720103be44b36a6f7d15307a437ff [id=12467795994e4dbaaa74e37c8b9cef32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.812558 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7a72db953646038a9d82dde04d76e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.813143 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 (DEFAULT_TABLE table=default.Table_415759429f02467f944d3c1a215d5966 [id=890bc27877e4475d8a573fa60c08055e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.813382 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.816190 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2776d8b893c94c099b63bb93030ca714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.816268 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d02d437359a643b59d1bd41dfe77a0fc (DEFAULT_TABLE table=default.Table_71248d2aeaee415194b2450bdb8def2e [id=f6381e088b094625be789e5010713590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.816335 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 9c90ec6864664fe48d8b9a92837b748c (DEFAULT_TABLE table=default.Table_3dd8aa3c2e914f019c0e7d29a3dbda13 [id=f55e6319c9174c39b0a8371818509111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.816504 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02d437359a643b59d1bd41dfe77a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.816605 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c90ec6864664fe48d8b9a92837b748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.817873 881 tablet_service.cc:1468] Processing CreateTablet for tablet 7edfdf771ecd4ae8b87791846536271f (DEFAULT_TABLE table=default.Table_0c86a89fdf674dfc856f5a107daad436 [id=af0787145f544902be2d758b2b957d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.818236 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edfdf771ecd4ae8b87791846536271f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.820286 1031 tablet_service.cc:1468] Processing CreateTablet for tablet bbfcc7513562418fa634b873beac2c4f (DEFAULT_TABLE table=default.Table_61af8dc03a3049d59036eab5d6240298 [id=ae30602e16274e1eb86d0a7013d8f249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.820381 882 tablet_service.cc:1468] Processing CreateTablet for tablet 69db100a14554271aee32c3f11af4346 (DEFAULT_TABLE table=default.Table_34b52be75f5d4681945cc00f5911c2d2 [id=562105e13fb64deebf82e76813e63002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.820540 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbfcc7513562418fa634b873beac2c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.820734 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69db100a14554271aee32c3f11af4346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.823979 893 tablet_service.cc:1468] Processing CreateTablet for tablet 2776d8b893c94c099b63bb93030ca714 (DEFAULT_TABLE table=default.Table_7bb26e59c27447b09a8cacbad6133b3a [id=0aadedc9390a4a28ad6c9db7c527eded]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.824254 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2776d8b893c94c099b63bb93030ca714. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.824381 894 tablet_service.cc:1468] Processing CreateTablet for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 (DEFAULT_TABLE table=default.Table_415759429f02467f944d3c1a215d5966 [id=890bc27877e4475d8a573fa60c08055e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.824667 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.827021 878 tablet_service.cc:1468] Processing CreateTablet for tablet ce7a72db953646038a9d82dde04d76e4 (DEFAULT_TABLE table=default.Table_3ec720103be44b36a6f7d15307a437ff [id=12467795994e4dbaaa74e37c8b9cef32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.827265 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7a72db953646038a9d82dde04d76e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.827538 891 tablet_service.cc:1468] Processing CreateTablet for tablet d02d437359a643b59d1bd41dfe77a0fc (DEFAULT_TABLE table=default.Table_71248d2aeaee415194b2450bdb8def2e [id=f6381e088b094625be789e5010713590]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.827780 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d02d437359a643b59d1bd41dfe77a0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.828303 887 tablet_service.cc:1468] Processing CreateTablet for tablet 9c90ec6864664fe48d8b9a92837b748c (DEFAULT_TABLE table=default.Table_3dd8aa3c2e914f019c0e7d29a3dbda13 [id=f55e6319c9174c39b0a8371818509111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.828549 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c90ec6864664fe48d8b9a92837b748c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.812507 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 7edfdf771ecd4ae8b87791846536271f (DEFAULT_TABLE table=default.Table_0c86a89fdf674dfc856f5a107daad436 [id=af0787145f544902be2d758b2b957d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.831187 883 tablet_service.cc:1468] Processing CreateTablet for tablet bbfcc7513562418fa634b873beac2c4f (DEFAULT_TABLE table=default.Table_61af8dc03a3049d59036eab5d6240298 [id=ae30602e16274e1eb86d0a7013d8f249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.831283 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edfdf771ecd4ae8b87791846536271f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.831420 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbfcc7513562418fa634b873beac2c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.836912 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce7a72db953646038a9d82dde04d76e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.852584 1801 raft_consensus.cc:491] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.852715 1801 raft_consensus.cc:513] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.853351 1801 leader_election.cc:290] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:49.854072 850 leader_election.cc:343] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 881461aa1bc34dd483f65e171c341bc5
I20250629 01:51:49.861068 1801 raft_consensus.cc:491] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.861197 1801 raft_consensus.cc:513] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.861740 1801 leader_election.cc:290] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:49.862416 850 leader_election.cc:343] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 96324ddf32a1460b8fc37e2c324649e6
W20250629 01:51:49.865424 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 476a6cf28db04f67bc23ddfc5cacad87 consensus state has no leader
I20250629 01:51:49.869449 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:49.869589 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:49.870695 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_21c4d86e1513448886bb2c76904e6ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.871536 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21c4d86e1513448886bb2c76904e6ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.872185 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c712d978a3848b9bb3abdbce8314dec with delete_type TABLET_DATA_DELETED (replaced by tablet 0323947b135a4308903b1eed1ca4db71) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.872378 1156 tablet_replica.cc:331] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:49.872532 1156 raft_consensus.cc:2241] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.872653 1156 raft_consensus.cc:2270] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.873282 1156 ts_tablet_manager.cc:1905] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.875500 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8a60215417c34bedafbf12172eab469a
I20250629 01:51:49.876261 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cf77f01c09db4169a6965146a7a98f10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:49.876439 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_632fd08e723c4862bb99e1041edbd220"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.876647 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf77f01c09db4169a6965146a7a98f10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:49.876791 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_632fd08e723c4862bb99e1041edbd220 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.877174 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 351f4a7ae7e244999dda81d282de505d with delete_type TABLET_DATA_DELETED (replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.877374 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a60215417c34bedafbf12172eab469a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.877712 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 43b845c3e7174618bd5c394d7206a634
W20250629 01:51:49.877820 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee7fb74854db4034a9d44f6d025e6756 consensus state has no leader
I20250629 01:51:49.878078 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_75440a0f96a3433f98a5840644b52d1f" } new_table_name: "default.Table_e50ddf6a53544cd59126c103fb46128f" modify_external_catalogs: true
I20250629 01:51:49.878309 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f9c2685e676b4fb386e49eb6b424c2a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.878652 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9c2685e676b4fb386e49eb6b424c2a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.879542 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 97d0fbb691514d51a1ca71f32a666c2d
I20250629 01:51:49.880182 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_727c4d417ecc455b9c30cc3618717c6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.880563 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_727c4d417ecc455b9c30cc3618717c6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:49.881417 850 leader_election.cc:343] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 881461aa1bc34dd483f65e171c341bc5
I20250629 01:51:49.881495 850 leader_election.cc:304] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:49.881796 850 leader_election.cc:343] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 96324ddf32a1460b8fc37e2c324649e6
I20250629 01:51:49.881870 850 leader_election.cc:304] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.882112 1801 raft_consensus.cc:2747] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.882258 1801 raft_consensus.cc:2747] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.882496 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a60215417c34bedafbf12172eab469a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.882714 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.883142 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.883344 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b845c3e7174618bd5c394d7206a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.883689 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 02bd46ceac1e4766ada4c4d2e95fda00
I20250629 01:51:49.884223 453 master_service.cc:948] ReplaceTablet: received request to replace tablet a6a012ba891949538800581436ef5380 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:49.884582 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.884770 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.884936 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b845c3e7174618bd5c394d7206a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.887037 1156 ts_tablet_manager.cc:1918] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.887213 1156 log.cc:1199] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0c712d978a3848b9bb3abdbce8314dec
I20250629 01:51:49.887616 1156 ts_tablet_manager.cc:1939] T 0c712d978a3848b9bb3abdbce8314dec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.888721 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0c712d978a3848b9bb3abdbce8314dec (table default.Table_ca2bcac172414e6c9f264cb2202a76fa [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:49.889070 1156 tablet_replica.cc:331] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:49.889317 1156 raft_consensus.cc:2241] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:49.889523 1156 raft_consensus.cc:2270] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:49.890298 1156 ts_tablet_manager.cc:1905] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:49.891109 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 75fd4ffcf896419b81a1c0eb7c5aeeec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.895188 1156 ts_tablet_manager.cc:1918] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.895304 1156 log.cc:1199] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/351f4a7ae7e244999dda81d282de505d
I20250629 01:51:49.895603 1156 ts_tablet_manager.cc:1939] T 351f4a7ae7e244999dda81d282de505d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:49.896879 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 351f4a7ae7e244999dda81d282de505d (table default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]) successfully deleted
W20250629 01:51:49.897333 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1ebbcb2c26b4667b2405154a9d8fccb consensus state has no leader
I20250629 01:51:49.897766 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet dc841eb171d04b118b1e705ceed4c861
I20250629 01:51:49.899789 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.899955 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.900076 892 tablet_service.cc:1515] Processing DeleteTablet for tablet dc841eb171d04b118b1e705ceed4c861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:49.900377 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76118a6e684448b09f244d44c909ccaa consensus state has no leader
I20250629 01:51:49.901430 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet dc841eb171d04b118b1e705ceed4c861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:49.902335 755 tablet_service.cc:1515] Processing DeleteTablet for tablet dc841eb171d04b118b1e705ceed4c861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:49.902947 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a60215417c34bedafbf12172eab469a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:49.903308 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b845c3e7174618bd5c394d7206a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:49.907742 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8a60215417c34bedafbf12172eab469a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a60215417c34bedafbf12172eab469a already in progress: creating tablet
W20250629 01:51:49.908803 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 97d0fbb691514d51a1ca71f32a666c2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d0fbb691514d51a1ca71f32a666c2d already in progress: creating tablet
W20250629 01:51:49.909400 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 43b845c3e7174618bd5c394d7206a634 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b845c3e7174618bd5c394d7206a634 already in progress: creating tablet
I20250629 01:51:49.911614 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 52c2dcaac80249f29effe1dacf3e6c8d (DEFAULT_TABLE table=default.Table_158439966f9d48839815a9c82d78c1ce [id=502592e797de4fc89529c49b57a3ae31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.911936 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c2dcaac80249f29effe1dacf3e6c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.912146 892 tablet_service.cc:1468] Processing CreateTablet for tablet a52ca6c256cc433d906e4733b80e3d76 (DEFAULT_TABLE table=default.Table_6217ce5beab343c0af71e61d491db6f8 [id=586642f2a286408a9fec9b54d584c603]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.912427 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52ca6c256cc433d906e4733b80e3d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.913035 746 tablet_service.cc:1468] Processing CreateTablet for tablet a52ca6c256cc433d906e4733b80e3d76 (DEFAULT_TABLE table=default.Table_6217ce5beab343c0af71e61d491db6f8 [id=586642f2a286408a9fec9b54d584c603]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.913302 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52ca6c256cc433d906e4733b80e3d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.915028 1015 tablet_service.cc:1468] Processing CreateTablet for tablet a52ca6c256cc433d906e4733b80e3d76 (DEFAULT_TABLE table=default.Table_6217ce5beab343c0af71e61d491db6f8 [id=586642f2a286408a9fec9b54d584c603]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.915419 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52ca6c256cc433d906e4733b80e3d76. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.918185 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 476a6cf28db04f67bc23ddfc5cacad87 (table default.Table_992098655a084dd1a8d522be5b7c1cc3 [id=b4d82748e8804c78a1bd48cef02e98a9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 476a6cf28db04f67bc23ddfc5cacad87 consensus state has no leader
I20250629 01:51:49.920302 1824 raft_consensus.cc:491] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.920414 1824 raft_consensus.cc:513] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.921003 1824 leader_election.cc:290] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:49.921476 1017 tablet_service.cc:1468] Processing CreateTablet for tablet e841a0c33e504a5081bae39d731106cb (DEFAULT_TABLE table=default.Table_7054a2ae6968462bbf1a604dc08243fc [id=c8b2adae52404d3da3564ac110ebeb3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.921545 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:49.921751 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841a0c33e504a5081bae39d731106cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.921729 1049 raft_consensus.cc:2466] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:49.921530 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee1e7985f4b4422ba607962bf1fdc42 (DEFAULT_TABLE table=default.Table_660af79d2b29409fae344ba080c4519a [id=f6910766df9448d2b29e4f0d2cee6650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.922001 754 tablet_service.cc:1468] Processing CreateTablet for tablet e841a0c33e504a5081bae39d731106cb (DEFAULT_TABLE table=default.Table_7054a2ae6968462bbf1a604dc08243fc [id=c8b2adae52404d3da3564ac110ebeb3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.922108 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee1e7985f4b4422ba607962bf1fdc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.922199 750 tablet_service.cc:1468] Processing CreateTablet for tablet c46bd4bfa331420abdcd5fe729300fc4 (DEFAULT_TABLE table=default.Table_57474f12b4c04c9c8a9e6e85838cdb8f [id=b51a24fe24f64bec8f075e07028d4b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.922279 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841a0c33e504a5081bae39d731106cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.922448 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46bd4bfa331420abdcd5fe729300fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.924275 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c46bd4bfa331420abdcd5fe729300fc4 (DEFAULT_TABLE table=default.Table_57474f12b4c04c9c8a9e6e85838cdb8f [id=b51a24fe24f64bec8f075e07028d4b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.924523 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46bd4bfa331420abdcd5fe729300fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.925694 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 71819b009efe45d8845af056f00b0535 (DEFAULT_TABLE table=default.Table_dc942d4ae58a4c82a26b62777b048719 [id=67926774e1cf40dcaa1a3d545e1bd625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.925935 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71819b009efe45d8845af056f00b0535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.927295 743 tablet_service.cc:1468] Processing CreateTablet for tablet e5e96c7df4b2470a89ef0ef8e9e97950 (DEFAULT_TABLE table=default.Table_c525edb71817495cbedf4ec757b7810e [id=b41bebebd46e4aa9a7aad0716ef7ad24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.927558 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e96c7df4b2470a89ef0ef8e9e97950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.928308 738 tablet_service.cc:1468] Processing CreateTablet for tablet 71819b009efe45d8845af056f00b0535 (DEFAULT_TABLE table=default.Table_dc942d4ae58a4c82a26b62777b048719 [id=67926774e1cf40dcaa1a3d545e1bd625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.928308 710 leader_election.cc:304] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:49.928580 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71819b009efe45d8845af056f00b0535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.928618 1824 raft_consensus.cc:2802] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:49.928684 1824 raft_consensus.cc:491] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:49.928746 1824 raft_consensus.cc:3058] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:49.931073 1824 raft_consensus.cc:513] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.931840 1824 leader_election.cc:290] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:49.931968 1824 raft_consensus.cc:491] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.932063 1824 raft_consensus.cc:513] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:49.932529 1824 leader_election.cc:290] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:49.933233 743 tablet_service.cc:1468] Processing CreateTablet for tablet f4a3b7d971694e15be1aaa072468dae2 (DEFAULT_TABLE table=default.Table_cbd53c3618ee416a842d119b503fa368 [id=092a23ac0eed43f4b09be946a09cc180]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.933493 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a3b7d971694e15be1aaa072468dae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.933650 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a8c3c036d11042159d012053bb2a29b3 (DEFAULT_TABLE table=default.Table_7c434928fb344af2aebbe8eb3f5376a2 [id=58e3615a7b6348b3a40d734406fdb0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.933928 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c3c036d11042159d012053bb2a29b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.934478 1710 raft_consensus.cc:491] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.934592 1710 raft_consensus.cc:513] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.935441 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f4a3b7d971694e15be1aaa072468dae2 (DEFAULT_TABLE table=default.Table_cbd53c3618ee416a842d119b503fa368 [id=092a23ac0eed43f4b09be946a09cc180]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.935531 740 tablet_service.cc:1468] Processing CreateTablet for tablet 51b8e24f87b34f15b45270d4693dc7b5 (DEFAULT_TABLE table=default.Table_d96fb8679e6648e0af1e38be2e2bc67f [id=310fe4ca0fa24fe787ecfb9c22dbbc95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.935635 1710 leader_election.cc:290] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:49.935698 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a3b7d971694e15be1aaa072468dae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.935786 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b8e24f87b34f15b45270d4693dc7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.936298 750 tablet_service.cc:1468] Processing CreateTablet for tablet a8c3c036d11042159d012053bb2a29b3 (DEFAULT_TABLE table=default.Table_7c434928fb344af2aebbe8eb3f5376a2 [id=58e3615a7b6348b3a40d734406fdb0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.936564 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c3c036d11042159d012053bb2a29b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.936084 746 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee1e7985f4b4422ba607962bf1fdc42 (DEFAULT_TABLE table=default.Table_660af79d2b29409fae344ba080c4519a [id=f6910766df9448d2b29e4f0d2cee6650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.941295 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee1e7985f4b4422ba607962bf1fdc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.943181 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e5e96c7df4b2470a89ef0ef8e9e97950 (DEFAULT_TABLE table=default.Table_c525edb71817495cbedf4ec757b7810e [id=b41bebebd46e4aa9a7aad0716ef7ad24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.943444 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e96c7df4b2470a89ef0ef8e9e97950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.943830 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 51b8e24f87b34f15b45270d4693dc7b5 (DEFAULT_TABLE table=default.Table_d96fb8679e6648e0af1e38be2e2bc67f [id=310fe4ca0fa24fe787ecfb9c22dbbc95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.944084 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b8e24f87b34f15b45270d4693dc7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.949234 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:49.950165 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:49.951018 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d85fd3d3dd4b07a812c019e1a5c748" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:49.951817 745 tablet_service.cc:1468] Processing CreateTablet for tablet bd09b5ae0dc44ce09638cd906400b38c (DEFAULT_TABLE table=default.Table_85166d71c7d74a25be9d12b625da1f5e [id=05eb7322226b4219a75c3ebdbdffed65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.951903 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543242e4a5064268ae1e5519b569f682" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:49.952299 1022 tablet_service.cc:1468] Processing CreateTablet for tablet bd09b5ae0dc44ce09638cd906400b38c (DEFAULT_TABLE table=default.Table_85166d71c7d74a25be9d12b625da1f5e [id=05eb7322226b4219a75c3ebdbdffed65]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:49.952409 986 leader_election.cc:343] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.952581 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd09b5ae0dc44ce09638cd906400b38c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:49.952795 712 leader_election.cc:343] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:49.953999 712 leader_election.cc:343] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:49.954877 712 leader_election.cc:343] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.954435 755 tablet_service.cc:1468] Processing CreateTablet for tablet 52c2dcaac80249f29effe1dacf3e6c8d (DEFAULT_TABLE table=default.Table_158439966f9d48839815a9c82d78c1ce [id=502592e797de4fc89529c49b57a3ae31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.956166 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd09b5ae0dc44ce09638cd906400b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.956362 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c2dcaac80249f29effe1dacf3e6c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.965595 1710 raft_consensus.cc:491] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:49.965718 1710 raft_consensus.cc:513] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:49.966246 1710 leader_election.cc:290] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:49.966933 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1fe3880e51345269e852014d0d60bb6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:49.967168 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 02bd46ceac1e4766ada4c4d2e95fda00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02bd46ceac1e4766ada4c4d2e95fda00 already in progress: creating tablet
W20250629 01:51:49.967374 988 leader_election.cc:343] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.967877 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1fe3880e51345269e852014d0d60bb6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:49.968317 986 leader_election.cc:343] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.968431 986 leader_election.cc:304] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:49.968601 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet dc841eb171d04b118b1e705ceed4c861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc841eb171d04b118b1e705ceed4c861 already in progress: creating tablet
I20250629 01:51:49.968775 1710 raft_consensus.cc:2747] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.969466 888 tablet_service.cc:1468] Processing CreateTablet for tablet 52c2dcaac80249f29effe1dacf3e6c8d (DEFAULT_TABLE table=default.Table_158439966f9d48839815a9c82d78c1ce [id=502592e797de4fc89529c49b57a3ae31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.969756 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "543242e4a5064268ae1e5519b569f682" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:49.972769 988 leader_election.cc:343] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:49.972868 988 leader_election.cc:304] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:49.973196 1710 raft_consensus.cc:2747] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:49.970088 890 tablet_service.cc:1468] Processing CreateTablet for tablet bd09b5ae0dc44ce09638cd906400b38c (DEFAULT_TABLE table=default.Table_85166d71c7d74a25be9d12b625da1f5e [id=05eb7322226b4219a75c3ebdbdffed65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:49.976583 1158 ts_tablet_manager.cc:1918] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.976797 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c2dcaac80249f29effe1dacf3e6c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.980193 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd09b5ae0dc44ce09638cd906400b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:49.985390 1157 ts_tablet_manager.cc:1918] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:49.991410 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6bc126479adc4caf8dbe1a49f8b14668"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:49.991904 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bc126479adc4caf8dbe1a49f8b14668 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.000450 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35873
--webserver_interface=127.0.104.61
--webserver_port=43231
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250629 01:51:50.008816 745 tablet_service.cc:1468] Processing CreateTablet for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 (DEFAULT_TABLE table=default.Table_cf77f01c09db4169a6965146a7a98f10 [id=58d87c5db5fa4954882510b24d6501c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.009130 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.012702 746 tablet_service.cc:1468] Processing CreateTablet for tablet 65592b4ef72f471481a275c11c780510 (DEFAULT_TABLE table=default.Table_f9c2685e676b4fb386e49eb6b424c2a6 [id=dc8496d93ce2446aa48b728e9edf7972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.012993 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65592b4ef72f471481a275c11c780510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.013850 740 tablet_service.cc:1468] Processing CreateTablet for tablet 03e4faf922e54c7a8bbe42694e0b899a (DEFAULT_TABLE table=default.Table_632fd08e723c4862bb99e1041edbd220 [id=0f112aabf0e34c7bb372a141c7374f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.014110 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e4faf922e54c7a8bbe42694e0b899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.012990 750 tablet_service.cc:1468] Processing CreateTablet for tablet 5a3752f8a275400da8cfff249f2dc6ee (DEFAULT_TABLE table=default.Table_b1aaca780fe54c5dbbba0d76d7194624 [id=24cb5273f5de4930a1843e69d423ebb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.016522 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3752f8a275400da8cfff249f2dc6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.012702 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5c1d273c70ac4db7a215f02ecac11154 (DEFAULT_TABLE table=default.Table_21c4d86e1513448886bb2c76904e6ef2 [id=9d8b48bfe2d64d53b022bbb8d1743eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.017405 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1d273c70ac4db7a215f02ecac11154. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.023382 740 tablet_service.cc:1468] Processing CreateTablet for tablet 8cea647451954d8ab3bd495a3a080c97 (DEFAULT_TABLE table=default.Table_966c88ce78e148fe9366b092be734d02 [id=db6bcc93c37d4a3b9bdbf8c92e897b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.023700 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cea647451954d8ab3bd495a3a080c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.024679 743 tablet_service.cc:1468] Processing CreateTablet for tablet 31354e63879041fa8a7a5d46e2712848 (DEFAULT_TABLE table=default.Table_727c4d417ecc455b9c30cc3618717c6c [id=c06daf8b99324d179c5e67049463a3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.024976 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31354e63879041fa8a7a5d46e2712848. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.037456 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d01934d1cb70450dbb1b6cf0c40c2129" } new_table_name: "default.Table_0fc926b866e240e592ab2c1505f115f5" modify_external_catalogs: true
I20250629 01:51:50.038858 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a526befbb9d14f6388686ec1248aa292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.039458 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a526befbb9d14f6388686ec1248aa292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.041606 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_92114ac6ae69479e86909ca0e6744dec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.042070 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92114ac6ae69479e86909ca0e6744dec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:50.043454 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2e9de48e4ad401880f351f2bd55cf5c consensus state has no leader
W20250629 01:51:50.047749 698 kernel_stack_watchdog.cc:198] Thread 1155 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 388ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f47c4642980 (unknown) at ??:0
@ 0x92ad77 __sanitizer::internal_sched_yield() at ??:?
@ 0x89042d __interceptor_pthread_create at ??:?
@ 0x7f47c12181d1 kudu::Thread::StartThread() at ??:0
@ 0xa621f7 kudu::Thread::Create() at ??:?
@ 0x7f47c123766b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f47c123308d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f47c12383bf kudu::ThreadPool::Submit() at ??:0
@ 0x7f47cc5bf5cd kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7f47cc5be5bf kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7f47cc5be17e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7f47cc5c61f5 kudu::log::Log::Init() at ??:0
@ 0x7f47cc5c5551 kudu::log::Log::Open() at ??:0
@ 0x7f47cdf05b61 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7f47cdf01ecc kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7f47cdf0064f kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20250629 01:51:50.010568 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 5c1d273c70ac4db7a215f02ecac11154 (DEFAULT_TABLE table=default.Table_21c4d86e1513448886bb2c76904e6ef2 [id=9d8b48bfe2d64d53b022bbb8d1743eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.057176 1158 log.cc:1199] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/77eae224763d464496066b4a98efe5ff
I20250629 01:51:50.057365 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1d273c70ac4db7a215f02ecac11154. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.057456 1158 ts_tablet_manager.cc:1939] T 77eae224763d464496066b4a98efe5ff P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.058220 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 77eae224763d464496066b4a98efe5ff (table default.Table_6f7d1b03023842daa7f9d43483dc6b95 [id=92be86eee6b64fb0984d8e5ea727fbdf]) successfully deleted
W20250629 01:51:50.058694 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ca13eb6bdcd044ec948ea699857a30f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca13eb6bdcd044ec948ea699857a30f8 already in progress: creating tablet
W20250629 01:51:50.059146 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f94d8fc1b6254ba69acadcaff3613d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f94d8fc1b6254ba69acadcaff3613d97 already in progress: creating tablet
W20250629 01:51:50.059590 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bdc6a26b3594d6fb33a27cac5b29f4f already in progress: creating tablet
I20250629 01:51:50.059791 1158 tablet_replica.cc:331] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.059938 1158 raft_consensus.cc:2241] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.060086 1158 raft_consensus.cc:2270] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.060524 1158 ts_tablet_manager.cc:1905] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.012256 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 5a3752f8a275400da8cfff249f2dc6ee (DEFAULT_TABLE table=default.Table_b1aaca780fe54c5dbbba0d76d7194624 [id=24cb5273f5de4930a1843e69d423ebb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.012349 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 (DEFAULT_TABLE table=default.Table_cf77f01c09db4169a6965146a7a98f10 [id=58d87c5db5fa4954882510b24d6501c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.013413 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 65592b4ef72f471481a275c11c780510 (DEFAULT_TABLE table=default.Table_f9c2685e676b4fb386e49eb6b424c2a6 [id=dc8496d93ce2446aa48b728e9edf7972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.023547 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 03e4faf922e54c7a8bbe42694e0b899a (DEFAULT_TABLE table=default.Table_632fd08e723c4862bb99e1041edbd220 [id=0f112aabf0e34c7bb372a141c7374f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.023536 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 8cea647451954d8ab3bd495a3a080c97 (DEFAULT_TABLE table=default.Table_966c88ce78e148fe9366b092be734d02 [id=db6bcc93c37d4a3b9bdbf8c92e897b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.031217 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 31354e63879041fa8a7a5d46e2712848 (DEFAULT_TABLE table=default.Table_727c4d417ecc455b9c30cc3618717c6c [id=c06daf8b99324d179c5e67049463a3dc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.045009 974 kernel_stack_watchdog.cc:198] Thread 1154 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 315ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f407328d980 (unknown) at ??:0
@ 0x92ad77 __sanitizer::internal_sched_yield() at ??:?
@ 0x89042d __interceptor_pthread_create at ??:?
@ 0x7f406fe631d1 kudu::Thread::StartThread() at ??:0
@ 0xa621f7 kudu::Thread::Create() at ??:?
@ 0x7f406fe8266b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f406fe7e08d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f406fe833bf kudu::ThreadPool::Submit() at ??:0
@ 0x7f407b20a5cd kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7f407b2095bf kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7f407b20917e kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7f407b2111f5 kudu::log::Log::Init() at ??:0
@ 0x7f407b210551 kudu::log::Log::Open() at ??:0
@ 0x7f407cb50b61 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7f407cb4cecc kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7f407cb4b64f kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20250629 01:51:50.048978 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:50.063160 1049 raft_consensus.cc:3058] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:50.065851 1049 raft_consensus.cc:2466] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:50.051890 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d85fd3d3dd4b07a812c019e1a5c748" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:50.066112 1048 raft_consensus.cc:2466] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:50.062676 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3752f8a275400da8cfff249f2dc6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.066402 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.071247 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31354e63879041fa8a7a5d46e2712848. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.073328 710 leader_election.cc:304] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:50.073752 1710 raft_consensus.cc:491] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.073848 1710 raft_consensus.cc:513] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.074354 1710 leader_election.cc:290] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:50.075119 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f0ae4c51924370896c624affde6761" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:50.075232 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e4faf922e54c7a8bbe42694e0b899a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:50.075512 986 leader_election.cc:343] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.075981 1850 raft_consensus.cc:2802] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:50.076226 1850 raft_consensus.cc:695] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:50.076480 1850 consensus_queue.cc:237] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.080893 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_13dc4660837b46219b404433b7272fce" } new_table_name: "default.Table_0cc47fb2736e4c40ba44dae6f0dec98d" modify_external_catalogs: true
W20250629 01:51:50.081099 1154 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.061s sys 0.126s
W20250629 01:51:50.081174 1154 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.561s user 0.061s sys 0.126s
W20250629 01:51:50.082412 1155 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.056s sys 0.131s
W20250629 01:51:50.082473 1155 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.561s user 0.056s sys 0.131s
I20250629 01:51:50.082542 710 leader_election.cc:304] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:50.082736 1824 raft_consensus.cc:2802] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:50.082787 1824 raft_consensus.cc:491] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:50.082840 1824 raft_consensus.cc:3058] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:50.083014 1154 tablet_bootstrap.cc:492] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.083108 1154 ts_tablet_manager.cc:1397] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.564s user 0.062s sys 0.126s
I20250629 01:51:50.083962 1154 raft_consensus.cc:357] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.084076 1154 raft_consensus.cc:383] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.084141 1154 raft_consensus.cc:738] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.084797 1155 tablet_bootstrap.cc:492] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.084914 1155 ts_tablet_manager.cc:1397] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.572s user 0.058s sys 0.131s
I20250629 01:51:50.085021 1824 raft_consensus.cc:513] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.085242 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6e4d952c729c4984a21f16da686e28f0" } modify_external_catalogs: true
I20250629 01:51:50.085381 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6e4d952c729c4984a21f16da686e28f0" } modify_external_catalogs: true
I20250629 01:51:50.085516 1824 leader_election.cc:290] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.085577 1155 raft_consensus.cc:357] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.085690 1155 raft_consensus.cc:383] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.085733 1155 raft_consensus.cc:738] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.085793 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_331d3b611f4f46748132e37fb1a8e872" } modify_external_catalogs: true
I20250629 01:51:50.085906 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_331d3b611f4f46748132e37fb1a8e872" } modify_external_catalogs: true
I20250629 01:51:50.085882 1155 consensus_queue.cc:260] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.086246 1155 ts_tablet_manager.cc:1428] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:50.086287 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_119824cff47b4becacd02655aba89e7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.086459 1155 tablet_bootstrap.cc:492] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:50.086721 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_119824cff47b4becacd02655aba89e7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.087625 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5da0f7f4f413447e84b57be34c67019b" } new_table_name: "default.Table_7ee7b9c60a0646c8addf07681430c183" modify_external_catalogs: true
I20250629 01:51:50.088068 1154 consensus_queue.cc:260] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.088661 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_992098655a084dd1a8d522be5b7c1cc3" } modify_external_catalogs: true
I20250629 01:51:50.088773 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_992098655a084dd1a8d522be5b7c1cc3" } modify_external_catalogs: true
I20250629 01:51:50.091688 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d96fb8679e6648e0af1e38be2e2bc67f" } modify_external_catalogs: true
I20250629 01:51:50.091754 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b16b6840c24484b90218fffc1da9621 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.091818 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d96fb8679e6648e0af1e38be2e2bc67f" } modify_external_catalogs: true
I20250629 01:51:50.092759 1154 ts_tablet_manager.cc:1428] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250629 01:51:50.095628 1154 tablet_bootstrap.cc:492] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:50.096370 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84cfa2d73c6b4ca2bf41437b196d10b1 consensus state has no leader
I20250629 01:51:50.096911 1154 tablet_bootstrap.cc:654] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.100757 1155 tablet_bootstrap.cc:654] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.102455 755 tablet_service.cc:1468] Processing CreateTablet for tablet f4cf919a96f34167b7dc9c3dfe7946a4 (DEFAULT_TABLE table=default.Table_6bc126479adc4caf8dbe1a49f8b14668 [id=bd70525dd2ec4973b2c7f3e992cdd80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.102744 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e26fc3ca9879424799115a5cb95e534c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.102864 1157 log.cc:1199] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5dfe921a84694903bbdb3fed3d6f2761
I20250629 01:51:50.102917 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8e4f4f0655b44c68953073e4f0298fb8" } modify_external_catalogs: true
I20250629 01:51:50.103057 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8e4f4f0655b44c68953073e4f0298fb8" } modify_external_catalogs: true
W20250629 01:51:50.103235 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e26fc3ca9879424799115a5cb95e534c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.103516 1157 ts_tablet_manager.cc:1939] T 5dfe921a84694903bbdb3fed3d6f2761 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:49.988379 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4ee1e7985f4b4422ba607962bf1fdc42 (DEFAULT_TABLE table=default.Table_660af79d2b29409fae344ba080c4519a [id=f6910766df9448d2b29e4f0d2cee6650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.102756 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cf919a96f34167b7dc9c3dfe7946a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.103829 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee1e7985f4b4422ba607962bf1fdc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.104146 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8a2b5b69e5db47f89ce6aafa5f16bd59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.104552 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a2b5b69e5db47f89ce6aafa5f16bd59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.990758 896 tablet_service.cc:1468] Processing CreateTablet for tablet 71819b009efe45d8845af056f00b0535 (DEFAULT_TABLE table=default.Table_dc942d4ae58a4c82a26b62777b048719 [id=67926774e1cf40dcaa1a3d545e1bd625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.106781 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71819b009efe45d8845af056f00b0535. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.107362 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_85166d71c7d74a25be9d12b625da1f5e" } new_table_name: "default.Table_bcfe0a9cbbf347619b9f9938b5d5943a" modify_external_catalogs: true
I20250629 01:51:50.109547 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4eb34771188b4b8d957400d4e3d8f3aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.109915 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4eb34771188b4b8d957400d4e3d8f3aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:49.992556 879 tablet_service.cc:1468] Processing CreateTablet for tablet e841a0c33e504a5081bae39d731106cb (DEFAULT_TABLE table=default.Table_7054a2ae6968462bbf1a604dc08243fc [id=c8b2adae52404d3da3564ac110ebeb3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.110517 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841a0c33e504a5081bae39d731106cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.110687 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cea647451954d8ab3bd495a3a080c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.009130 886 tablet_service.cc:1468] Processing CreateTablet for tablet c46bd4bfa331420abdcd5fe729300fc4 (DEFAULT_TABLE table=default.Table_57474f12b4c04c9c8a9e6e85838cdb8f [id=b51a24fe24f64bec8f075e07028d4b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.113219 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46bd4bfa331420abdcd5fe729300fc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.010358 895 tablet_service.cc:1468] Processing CreateTablet for tablet e5e96c7df4b2470a89ef0ef8e9e97950 (DEFAULT_TABLE table=default.Table_c525edb71817495cbedf4ec757b7810e [id=b41bebebd46e4aa9a7aad0716ef7ad24]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.116389 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e96c7df4b2470a89ef0ef8e9e97950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.011322 884 tablet_service.cc:1468] Processing CreateTablet for tablet a8c3c036d11042159d012053bb2a29b3 (DEFAULT_TABLE table=default.Table_7c434928fb344af2aebbe8eb3f5376a2 [id=58e3615a7b6348b3a40d734406fdb0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.120039 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c3c036d11042159d012053bb2a29b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.041085 1159 tablet_bootstrap.cc:492] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:50.123099 1159 ts_tablet_manager.cc:1397] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.340s user 0.000s sys 0.004s
I20250629 01:51:50.123680 1159 raft_consensus.cc:357] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.124668 1159 raft_consensus.cc:383] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.124747 1159 raft_consensus.cc:738] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.125200 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0ba9eabd06174b87a1485f4921895c5f
I20250629 01:51:50.125329 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 51b8e24f87b34f15b45270d4693dc7b5
W20250629 01:51:50.125922 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 621abed696cd4284bf8ee0223df312ee consensus state has no leader
I20250629 01:51:50.126067 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 40592f11f430469f9c8e82eba25b38ed
I20250629 01:51:50.126025 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4b91bc6605e44d8ea93074772ff7afa8" } new_table_name: "default.Table_92526d1aecd94064817e09aca50dd8d0" modify_external_catalogs: true
I20250629 01:51:50.126166 447 master_service.cc:948] ReplaceTablet: received request to replace tablet c40a07f50971493cb353ed94e850719a from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.126617 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c74c8328d4ae482196d5c984598ad171"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.126988 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c74c8328d4ae482196d5c984598ad171 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.127506 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ba9eabd06174b87a1485f4921895c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.125941 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_69b1cdc314154738a9f8f680f6c6bd40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.127869 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 51b8e24f87b34f15b45270d4693dc7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.128031 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f4cf919a96f34167b7dc9c3dfe7946a4 (DEFAULT_TABLE table=default.Table_6bc126479adc4caf8dbe1a49f8b14668 [id=bd70525dd2ec4973b2c7f3e992cdd80f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.128152 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b1cdc314154738a9f8f680f6c6bd40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.128132 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 40592f11f430469f9c8e82eba25b38ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.128144 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ba9eabd06174b87a1485f4921895c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.128367 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cf919a96f34167b7dc9c3dfe7946a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.125025 1159 consensus_queue.cc:260] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.128894 1159 ts_tablet_manager.cc:1428] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:50.075807 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f0ae4c51924370896c624affde6761" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.090531 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d85fd3d3dd4b07a812c019e1a5c748" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:50.129329 712 leader_election.cc:343] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:50.129631 988 leader_election.cc:343] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.129743 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 476a6cf28db04f67bc23ddfc5cacad87
I20250629 01:51:50.129729 988 leader_election.cc:304] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.129930 1710 raft_consensus.cc:2747] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.130275 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 40592f11f430469f9c8e82eba25b38ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.130256 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9abd434ca0b940478c9661546ed480a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.130440 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 51b8e24f87b34f15b45270d4693dc7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:50.130620 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9abd434ca0b940478c9661546ed480a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:50.130672 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 57001us
I20250629 01:51:50.131107 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26d85fd3d3dd4b07a812c019e1a5c748" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:50.131237 1048 raft_consensus.cc:3058] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:50.131757 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 51b8e24f87b34f15b45270d4693dc7b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b8e24f87b34f15b45270d4693dc7b5 already in progress: creating tablet
W20250629 01:51:50.131949 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 40592f11f430469f9c8e82eba25b38ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40592f11f430469f9c8e82eba25b38ed already in progress: creating tablet
I20250629 01:51:50.132222 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65592b4ef72f471481a275c11c780510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.132493 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:50.133330 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0ba9eabd06174b87a1485f4921895c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ba9eabd06174b87a1485f4921895c5f already in progress: creating tablet
W20250629 01:51:50.133662 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 476a6cf28db04f67bc23ddfc5cacad87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476a6cf28db04f67bc23ddfc5cacad87 already in progress: creating tablet
I20250629 01:51:50.133756 1048 raft_consensus.cc:2466] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:50.134541 710 leader_election.cc:304] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:50.134749 1824 raft_consensus.cc:2802] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:50.135227 1154 tablet_bootstrap.cc:492] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.135346 1154 ts_tablet_manager.cc:1397] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.006s
I20250629 01:51:50.136746 1155 tablet_bootstrap.cc:492] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.136844 1155 ts_tablet_manager.cc:1397] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.003s
I20250629 01:51:50.137459 1155 raft_consensus.cc:357] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.137619 1155 raft_consensus.cc:383] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.137670 1155 raft_consensus.cc:738] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.137836 1155 consensus_queue.cc:260] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.138130 1154 raft_consensus.cc:357] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.138283 1154 raft_consensus.cc:383] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.138324 1154 raft_consensus.cc:738] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.139703 1154 consensus_queue.cc:260] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.141674 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.141894 1154 ts_tablet_manager.cc:1428] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:50.142095 1154 tablet_bootstrap.cc:492] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.142261 894 tablet_service.cc:1468] Processing CreateTablet for tablet 51b8e24f87b34f15b45270d4693dc7b5 (DEFAULT_TABLE table=default.Table_d96fb8679e6648e0af1e38be2e2bc67f [id=310fe4ca0fa24fe787ecfb9c22dbbc95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.142556 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b8e24f87b34f15b45270d4693dc7b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.144336 1155 ts_tablet_manager.cc:1428] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20250629 01:51:50.144537 1155 tablet_bootstrap.cc:492] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.145804 1155 tablet_bootstrap.cc:654] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.146335 882 tablet_service.cc:1468] Processing CreateTablet for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 (DEFAULT_TABLE table=default.Table_cf77f01c09db4169a6965146a7a98f10 [id=58d87c5db5fa4954882510b24d6501c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.146607 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.147431 1154 tablet_bootstrap.cc:654] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.147718 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 26d85fd3d3dd4b07a812c019e1a5c748
W20250629 01:51:50.148531 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd09b5ae0dc44ce09638cd906400b38c consensus state has no leader
I20250629 01:51:50.150861 887 tablet_service.cc:1468] Processing CreateTablet for tablet 5c1d273c70ac4db7a215f02ecac11154 (DEFAULT_TABLE table=default.Table_21c4d86e1513448886bb2c76904e6ef2 [id=9d8b48bfe2d64d53b022bbb8d1743eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.151165 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1d273c70ac4db7a215f02ecac11154. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.154978 883 tablet_service.cc:1468] Processing CreateTablet for tablet 5a3752f8a275400da8cfff249f2dc6ee (DEFAULT_TABLE table=default.Table_b1aaca780fe54c5dbbba0d76d7194624 [id=24cb5273f5de4930a1843e69d423ebb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.155261 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a3752f8a275400da8cfff249f2dc6ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.156164 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c40a07f50971493cb353ed94e850719a
I20250629 01:51:50.156502 447 catalog_manager.cc:6652] ReplaceTablet: tablet c40a07f50971493cb353ed94e850719a deleted and replaced by tablet 40266441c3684726a58a5029779395fc
W20250629 01:51:50.157078 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac9d77381f7e4241aa546eaea3077168 consensus state has no leader
I20250629 01:51:50.158376 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c40a07f50971493cb353ed94e850719a with delete_type TABLET_DATA_DELETED (replaced by tablet 40266441c3684726a58a5029779395fc) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.158653 891 tablet_service.cc:1468] Processing CreateTablet for tablet 65592b4ef72f471481a275c11c780510 (DEFAULT_TABLE table=default.Table_f9c2685e676b4fb386e49eb6b424c2a6 [id=dc8496d93ce2446aa48b728e9edf7972]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.158948 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65592b4ef72f471481a275c11c780510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.158944 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet c40a07f50971493cb353ed94e850719a with delete_type TABLET_DATA_DELETED (replaced by tablet 40266441c3684726a58a5029779395fc) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:50.159773 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c40a07f50971493cb353ed94e850719a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40a07f50971493cb353ed94e850719a already in progress: creating tablet
I20250629 01:51:50.162674 878 tablet_service.cc:1468] Processing CreateTablet for tablet 03e4faf922e54c7a8bbe42694e0b899a (DEFAULT_TABLE table=default.Table_632fd08e723c4862bb99e1041edbd220 [id=0f112aabf0e34c7bb372a141c7374f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.162925 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03e4faf922e54c7a8bbe42694e0b899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.165968 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.166203 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.166363 1156 tablet_replica.cc:331] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.166698 892 tablet_service.cc:1468] Processing CreateTablet for tablet 8cea647451954d8ab3bd495a3a080c97 (DEFAULT_TABLE table=default.Table_966c88ce78e148fe9366b092be734d02 [id=db6bcc93c37d4a3b9bdbf8c92e897b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.166970 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cea647451954d8ab3bd495a3a080c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.176280 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.174644 881 tablet_service.cc:1468] Processing CreateTablet for tablet f4cf919a96f34167b7dc9c3dfe7946a4 (DEFAULT_TABLE table=default.Table_6bc126479adc4caf8dbe1a49f8b14668 [id=bd70525dd2ec4973b2c7f3e992cdd80f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.181970 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cf919a96f34167b7dc9c3dfe7946a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.183851 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 40592f11f430469f9c8e82eba25b38ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.183983 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.183974 1850 raft_consensus.cc:491] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.184187 1850 raft_consensus.cc:513] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.184695 1850 leader_election.cc:290] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.176291 893 tablet_service.cc:1468] Processing CreateTablet for tablet f4a3b7d971694e15be1aaa072468dae2 (DEFAULT_TABLE table=default.Table_cbd53c3618ee416a842d119b503fa368 [id=092a23ac0eed43f4b09be946a09cc180]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.187835 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a3b7d971694e15be1aaa072468dae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.176455 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ba9eabd06174b87a1485f4921895c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.192721 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.192857 1158 ts_tablet_manager.cc:1918] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.192934 1158 log.cc:1199] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e1fe3880e51345269e852014d0d60bb6
W20250629 01:51:50.193104 712 leader_election.cc:343] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.193158 1158 ts_tablet_manager.cc:1939] T e1fe3880e51345269e852014d0d60bb6 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.171825 892 tablet_service.cc:1468] Processing CreateTablet for tablet 31354e63879041fa8a7a5d46e2712848 (DEFAULT_TABLE table=default.Table_727c4d417ecc455b9c30cc3618717c6c [id=c06daf8b99324d179c5e67049463a3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.180402 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 51b8e24f87b34f15b45270d4693dc7b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.193851 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 707) took 1049 ms. Trace:
I20250629 01:51:50.193931 1158 rpcz_store.cc:276] 0629 01:51:49.144562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.183517 (+ 38955us) service_pool.cc:225] Handling call
0629 01:51:50.059757 (+876240us) ts_tablet_manager.cc:1145] Deleting tablet e1fe3880e51345269e852014d0d60bb6
0629 01:51:50.059764 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.062913 (+ 3149us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.192776 (+129863us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.193830 (+ 1054us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":873519}
I20250629 01:51:50.194422 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e1fe3880e51345269e852014d0d60bb6 (table default.Table_cf2dc25aa17a42fb91e6903713ca0d6c [id=4e38782b406e4cd9ae3278d074a1d97e]) successfully deleted
I20250629 01:51:50.194666 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31354e63879041fa8a7a5d46e2712848. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.196468 1159 tablet_bootstrap.cc:492] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:50.197233 755 tablet_service.cc:1468] Processing CreateTablet for tablet 31cee0e741584157b0333851f94c8559 (DEFAULT_TABLE table=default.Table_119824cff47b4becacd02655aba89e7f [id=b32e4a7122a64755a4eb0ca214930399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.197535 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31cee0e741584157b0333851f94c8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.197779 1159 tablet_bootstrap.cc:654] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:50.198989 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 64458.8us
W20250629 01:51:50.200716 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 107627us
I20250629 01:51:50.212396 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 660) took 1330 ms. Trace:
I20250629 01:51:50.201360 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:50.340750 1048 raft_consensus.cc:2466] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:51:50.206000 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35873, user_credentials={real_user=slave}} blocked reactor thread for 74448.8us
I20250629 01:51:50.202876 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 31cee0e741584157b0333851f94c8559 (DEFAULT_TABLE table=default.Table_119824cff47b4becacd02655aba89e7f [id=b32e4a7122a64755a4eb0ca214930399]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.344372 440 catalog_manager.cc:5001] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): alter failed for tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]),no further retry: Not found: Tablet not found: bfa48f94ac0e48e39578a6e8af5c8e9c
I20250629 01:51:50.344466 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31cee0e741584157b0333851f94c8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.223069 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 44ab886d05e0468ab0e7b156a32dd392 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.205797 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 18a5af7b1c8b40dd8751d5d7a694c3eb (DEFAULT_TABLE table=default.Table_a526befbb9d14f6388686ec1248aa292 [id=98492626be3b4c4487e6170d1239278c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.206766 1154 tablet_bootstrap.cc:492] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.348845 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a5af7b1c8b40dd8751d5d7a694c3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.348886 1154 ts_tablet_manager.cc:1397] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.207s user 0.003s sys 0.009s
I20250629 01:51:50.349525 1154 raft_consensus.cc:357] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.349654 1154 raft_consensus.cc:383] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.349691 1154 raft_consensus.cc:738] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.349843 1154 consensus_queue.cc:260] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.204715 743 tablet_service.cc:1468] Processing CreateTablet for tablet 18a5af7b1c8b40dd8751d5d7a694c3eb (DEFAULT_TABLE table=default.Table_a526befbb9d14f6388686ec1248aa292 [id=98492626be3b4c4487e6170d1239278c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.216708 1159 tablet_bootstrap.cc:492] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:50.205487 1824 raft_consensus.cc:695] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:50.220649 1157 rpcz_store.cc:276] 0629 01:51:48.882041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.886262 (+ 4221us) service_pool.cc:225] Handling call
0629 01:51:49.661671 (+775409us) ts_tablet_manager.cc:1145] Deleting tablet 5dfe921a84694903bbdb3fed3d6f2761
0629 01:51:49.661681 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:49.753773 (+ 92092us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:49.985320 (+231547us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.212379 (+227059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":775125}
I20250629 01:51:50.387159 1159 ts_tablet_manager.cc:1397] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.191s user 0.001s sys 0.001s
I20250629 01:51:50.387272 1157 tablet_replica.cc:331] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:50.387391 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a5af7b1c8b40dd8751d5d7a694c3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.387429 1157 raft_consensus.cc:2241] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.387302 1824 consensus_queue.cc:237] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.387581 1157 raft_consensus.cc:2270] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.387909 1159 raft_consensus.cc:357] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.388038 1159 raft_consensus.cc:383] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.388065 1157 ts_tablet_manager.cc:1905] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.388458 1156 raft_consensus.cc:2241] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:50.223711 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6e9154fa65874eb7adab5dcc6ca51d32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.235246 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a692f00ba244a80b9a6d07029f8d27e from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:50.391572 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e9154fa65874eb7adab5dcc6ca51d32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.350128 1154 ts_tablet_manager.cc:1428] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:50.244827 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_860e06a9808a4f8d9be0cedf70de1be4" } new_table_name: "default.Table_1777ed0cebb04529b9bdc951af9addf4" modify_external_catalogs: true
I20250629 01:51:50.207538 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 9e0d550773c949df9992be1f9f01b548 (DEFAULT_TABLE table=default.Table_92114ac6ae69479e86909ca0e6744dec [id=17b2a91306024c68aa06426b022f6111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.247964 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.394665 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.244012 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 845f2422d6494c95a4738f3062c216c7 (DEFAULT_TABLE table=default.Table_9abd434ca0b940478c9661546ed480a2 [id=68eb8cc1cacd46cf8bd81fb871c5d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.251997 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a092a591616041599f1b07e758ef38d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.395853 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a092a591616041599f1b07e758ef38d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.244308 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 109e19ed0ea64fa4985e93efbaca5107 (DEFAULT_TABLE table=default.Table_c74c8328d4ae482196d5c984598ad171 [id=634929efbe3b409d83cc43db0e067cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.253096 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 69980d5bf37d4001a6196b22c34ccc94 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.253199 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 9b54228f0013424588a355ceb17d9784 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.397424 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af7702628dd2458d831dcc0ce1be9327" } modify_external_catalogs: true
I20250629 01:51:50.297750 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dfa935fc642246518e47197bd7dda059" } new_table_name: "default.Table_71d82a98d0124d08a3ac0fce4fb47062" modify_external_catalogs: true
I20250629 01:51:50.397630 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af7702628dd2458d831dcc0ce1be9327" } modify_external_catalogs: true
I20250629 01:51:50.298743 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_32c0db573e334edd9afb9efcdff8d7ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.244588 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 13ee987a9e3e4f19a99e07cb878b7511 (DEFAULT_TABLE table=default.Table_69b1cdc314154738a9f8f680f6c6bd40 [id=5f256903bce841b5acdff39d8082db4b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.398739 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32c0db573e334edd9afb9efcdff8d7ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.245762 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ec0eeb0ef02c416abc5665f058c6bd7f (DEFAULT_TABLE table=default.Table_8a2b5b69e5db47f89ce6aafa5f16bd59 [id=daa2753835214cdea152ac8b4747c5ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.245872 1021 tablet_service.cc:1468] Processing CreateTablet for tablet bf110bfaab6948f3a32cdf45241b31a7 (DEFAULT_TABLE table=default.Table_4eb34771188b4b8d957400d4e3d8f3aa [id=00c755e2c2ff4b5f9790e66b12f5b790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.246124 1032 tablet_service.cc:1468] Processing CreateTablet for tablet fc67c212516748ae99c1c8e51313aecf (DEFAULT_TABLE table=default.Table_e26fc3ca9879424799115a5cb95e534c [id=06e0e67f5af244d29fa32f1a6488b598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.246353 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 40266441c3684726a58a5029779395fc (DEFAULT_TABLE table=default.Table_d9cb1a00d22843f69edc93ba305abaad [id=6153bb5c750f4685a0bd20a49447ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.389024 1156 pending_rounds.cc:70] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:50.399117 1156 pending_rounds.cc:77] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759184948817920 op_type: NO_OP noop_request { }
I20250629 01:51:50.399225 1156 raft_consensus.cc:2887] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:50.399300 1156 raft_consensus.cc:2270] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.223299 1155 tablet_bootstrap.cc:492] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.399526 1155 ts_tablet_manager.cc:1397] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.255s user 0.005s sys 0.006s
I20250629 01:51:50.207391 1158 tablet_replica.cc:331] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.356649 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 44ab886d05e0468ab0e7b156a32dd392
I20250629 01:51:50.400167 1155 raft_consensus.cc:357] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.400271 451 catalog_manager.cc:6652] ReplaceTablet: tablet 44ab886d05e0468ab0e7b156a32dd392 deleted and replaced by tablet f4609264b54241a0af744585f8bdbc89
I20250629 01:51:50.400285 1155 raft_consensus.cc:383] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.400321 1155 raft_consensus.cc:738] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.400467 1155 consensus_queue.cc:260] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.400571 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_eb9a6975124a4d049fa2c7043760c952" } new_table_name: "default.Table_67f686028f0c40c991d3231bd0b0c823" modify_external_catalogs: true
I20250629 01:51:50.400848 1155 ts_tablet_manager.cc:1428] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:50.401078 1155 tablet_bootstrap.cc:492] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.401060 1158 raft_consensus.cc:2241] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.401203 1158 raft_consensus.cc:2270] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.390476 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 5dfe921a84694903bbdb3fed3d6f2761 (table default.Table_6ada0762e0224924acf20a4507ad2fa6 [id=82f79023b3c0425c978152033e2183b3]) successfully deleted
I20250629 01:51:50.242818 746 tablet_service.cc:1468] Processing CreateTablet for tablet bf110bfaab6948f3a32cdf45241b31a7 (DEFAULT_TABLE table=default.Table_4eb34771188b4b8d957400d4e3d8f3aa [id=00c755e2c2ff4b5f9790e66b12f5b790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.401701 1158 ts_tablet_manager.cc:1905] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.401906 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf110bfaab6948f3a32cdf45241b31a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.402379 1155 tablet_bootstrap.cc:654] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.243165 740 tablet_service.cc:1468] Processing CreateTablet for tablet 845f2422d6494c95a4738f3062c216c7 (DEFAULT_TABLE table=default.Table_9abd434ca0b940478c9661546ed480a2 [id=68eb8cc1cacd46cf8bd81fb871c5d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.404461 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845f2422d6494c95a4738f3062c216c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.243474 750 tablet_service.cc:1468] Processing CreateTablet for tablet 109e19ed0ea64fa4985e93efbaca5107 (DEFAULT_TABLE table=default.Table_c74c8328d4ae482196d5c984598ad171 [id=634929efbe3b409d83cc43db0e067cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.243709 745 tablet_service.cc:1468] Processing CreateTablet for tablet 13ee987a9e3e4f19a99e07cb878b7511 (DEFAULT_TABLE table=default.Table_69b1cdc314154738a9f8f680f6c6bd40 [id=5f256903bce841b5acdff39d8082db4b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.408439 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f9f59fbe9634ac8b27f990d9496f637 consensus state has no leader
I20250629 01:51:50.408478 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ee987a9e3e4f19a99e07cb878b7511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.341584 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.246850 738 tablet_service.cc:1468] Processing CreateTablet for tablet ec0eeb0ef02c416abc5665f058c6bd7f (DEFAULT_TABLE table=default.Table_8a2b5b69e5db47f89ce6aafa5f16bd59 [id=daa2753835214cdea152ac8b4747c5ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.409176 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0eeb0ef02c416abc5665f058c6bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.409624 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f5ba36b31e8a415b81ec003950152c56" } modify_external_catalogs: true
I20250629 01:51:50.409775 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f5ba36b31e8a415b81ec003950152c56" } modify_external_catalogs: true
I20250629 01:51:50.341861 886 tablet_service.cc:1468] Processing CreateTablet for tablet 31cee0e741584157b0333851f94c8559 (DEFAULT_TABLE table=default.Table_119824cff47b4becacd02655aba89e7f [id=b32e4a7122a64755a4eb0ca214930399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.410210 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31cee0e741584157b0333851f94c8559. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.247049 754 tablet_service.cc:1468] Processing CreateTablet for tablet fc67c212516748ae99c1c8e51313aecf (DEFAULT_TABLE table=default.Table_e26fc3ca9879424799115a5cb95e534c [id=06e0e67f5af244d29fa32f1a6488b598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.415378 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc67c212516748ae99c1c8e51313aecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.342233 884 tablet_service.cc:1468] Processing CreateTablet for tablet 18a5af7b1c8b40dd8751d5d7a694c3eb (DEFAULT_TABLE table=default.Table_a526befbb9d14f6388686ec1248aa292 [id=98492626be3b4c4487e6170d1239278c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.416075 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a5af7b1c8b40dd8751d5d7a694c3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.416154 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 44ab886d05e0468ab0e7b156a32dd392 with delete_type TABLET_DATA_DELETED (replaced by tablet f4609264b54241a0af744585f8bdbc89) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.417028 454 catalog_manager.cc:5582] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:50.417364 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:50.247189 749 tablet_service.cc:1468] Processing CreateTablet for tablet 40266441c3684726a58a5029779395fc (DEFAULT_TABLE table=default.Table_d9cb1a00d22843f69edc93ba305abaad [id=6153bb5c750f4685a0bd20a49447ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.420506 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40266441c3684726a58a5029779395fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.420624 1824 raft_consensus.cc:491] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.420739 1824 raft_consensus.cc:513] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.342882 885 tablet_service.cc:1468] Processing CreateTablet for tablet 9e0d550773c949df9992be1f9f01b548 (DEFAULT_TABLE table=default.Table_92114ac6ae69479e86909ca0e6744dec [id=17b2a91306024c68aa06426b022f6111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.421113 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0d550773c949df9992be1f9f01b548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.343295 710 leader_election.cc:304] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:51:50.424374 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 81112us
I20250629 01:51:50.341594 895 tablet_service.cc:1515] Processing DeleteTablet for tablet c40a07f50971493cb353ed94e850719a with delete_type TABLET_DATA_DELETED (replaced by tablet 40266441c3684726a58a5029779395fc) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.351850 890 tablet_service.cc:1468] Processing CreateTablet for tablet 845f2422d6494c95a4738f3062c216c7 (DEFAULT_TABLE table=default.Table_9abd434ca0b940478c9661546ed480a2 [id=68eb8cc1cacd46cf8bd81fb871c5d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.424630 1850 raft_consensus.cc:2802] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:50.424713 1850 raft_consensus.cc:491] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:50.424826 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845f2422d6494c95a4738f3062c216c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.424809 1850 raft_consensus.cc:3058] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:50.427704 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0d550773c949df9992be1f9f01b548. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.351974 896 tablet_service.cc:1468] Processing CreateTablet for tablet 109e19ed0ea64fa4985e93efbaca5107 (DEFAULT_TABLE table=default.Table_c74c8328d4ae482196d5c984598ad171 [id=634929efbe3b409d83cc43db0e067cb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.427711 1154 tablet_bootstrap.cc:492] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.429982 1154 tablet_bootstrap.cc:654] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.352269 879 tablet_service.cc:1468] Processing CreateTablet for tablet 13ee987a9e3e4f19a99e07cb878b7511 (DEFAULT_TABLE table=default.Table_69b1cdc314154738a9f8f680f6c6bd40 [id=5f256903bce841b5acdff39d8082db4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.365382 1710 raft_consensus.cc:491] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.431388 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9b54228f0013424588a355ceb17d9784
I20250629 01:51:50.431423 1710 raft_consensus.cc:513] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.431813 448 catalog_manager.cc:6652] ReplaceTablet: tablet 9b54228f0013424588a355ceb17d9784 deleted and replaced by tablet ac560295fddd44368e6f9bc543f7cbfa
I20250629 01:51:50.431941 1710 leader_election.cc:290] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:50.432263 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845f2422d6494c95a4738f3062c216c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.432873 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_951aa64d945448d4b3c17187cf5bff48" } modify_external_catalogs: true
I20250629 01:51:50.433001 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_951aa64d945448d4b3c17187cf5bff48" } modify_external_catalogs: true
I20250629 01:51:50.433425 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b92827e379f148d4bd76b3e578adc695"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.433822 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b92827e379f148d4bd76b3e578adc695 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:50.434759 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a53d9e8b7efe46099821ae628640a42e consensus state has no leader
I20250629 01:51:50.435007 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8bf46b174da647d2b1135f050ce5e430" } modify_external_catalogs: true
I20250629 01:51:50.435140 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8bf46b174da647d2b1135f050ce5e430" } modify_external_catalogs: true
I20250629 01:51:50.352520 888 tablet_service.cc:1468] Processing CreateTablet for tablet bf110bfaab6948f3a32cdf45241b31a7 (DEFAULT_TABLE table=default.Table_4eb34771188b4b8d957400d4e3d8f3aa [id=00c755e2c2ff4b5f9790e66b12f5b790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.436060 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a792a26725954adea15ad57e31bfb224" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:50.436520 986 leader_election.cc:343] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.352768 889 tablet_service.cc:1468] Processing CreateTablet for tablet ec0eeb0ef02c416abc5665f058c6bd7f (DEFAULT_TABLE table=default.Table_8a2b5b69e5db47f89ce6aafa5f16bd59 [id=daa2753835214cdea152ac8b4747c5ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.437813 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109e19ed0ea64fa4985e93efbaca5107. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.353019 878 tablet_service.cc:1468] Processing CreateTablet for tablet fc67c212516748ae99c1c8e51313aecf (DEFAULT_TABLE table=default.Table_e26fc3ca9879424799115a5cb95e534c [id=06e0e67f5af244d29fa32f1a6488b598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.439483 1158 ts_tablet_manager.cc:1918] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.439565 1158 log.cc:1199] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0827ec622ff04ca6b2fec50e40b9e1de
I20250629 01:51:50.439805 1158 ts_tablet_manager.cc:1939] T 0827ec622ff04ca6b2fec50e40b9e1de P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.353262 881 tablet_service.cc:1468] Processing CreateTablet for tablet 40266441c3684726a58a5029779395fc (DEFAULT_TABLE table=default.Table_d9cb1a00d22843f69edc93ba305abaad [id=6153bb5c750f4685a0bd20a49447ba87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.440718 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a792a26725954adea15ad57e31bfb224" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.442414 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b54228f0013424588a355ceb17d9784 with delete_type TABLET_DATA_DELETED (replaced by tablet ac560295fddd44368e6f9bc543f7cbfa) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.442529 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109e19ed0ea64fa4985e93efbaca5107. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.443360 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ee987a9e3e4f19a99e07cb878b7511. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:50.444249 988 leader_election.cc:343] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.444342 988 leader_election.cc:304] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.448299 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0eeb0ef02c416abc5665f058c6bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.388077 1159 raft_consensus.cc:738] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.393693 1157 ts_tablet_manager.cc:1918] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:50.404471 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.404587 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 44ab886d05e0468ab0e7b156a32dd392 with delete_type TABLET_DATA_DELETED (replaced by tablet f4609264b54241a0af744585f8bdbc89) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.450982 1159 consensus_queue.cc:260] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.450995 1157 log.cc:1199] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/bfa48f94ac0e48e39578a6e8af5c8e9c
I20250629 01:51:50.451428 1159 ts_tablet_manager.cc:1428] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.064s user 0.001s sys 0.001s
I20250629 01:51:50.451498 1157 ts_tablet_manager.cc:1939] T bfa48f94ac0e48e39578a6e8af5c8e9c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:50.452126 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b1aaca780fe54c5dbbba0d76d7194624" } modify_external_catalogs: true
I20250629 01:51:50.452286 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf110bfaab6948f3a32cdf45241b31a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.452279 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b1aaca780fe54c5dbbba0d76d7194624" } modify_external_catalogs: true
I20250629 01:51:50.453522 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7ee7b9c60a0646c8addf07681430c183" } new_table_name: "default.Table_cd0e5afd755c4cdda905a0e1073f0595" modify_external_catalogs: true
I20250629 01:51:50.454396 1710 raft_consensus.cc:2747] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.455248 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 658) took 1590 ms. Trace:
I20250629 01:51:50.455312 1157 rpcz_store.cc:276] 0629 01:51:48.864846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.895542 (+ 30696us) service_pool.cc:225] Handling call
0629 01:51:50.387226 (+1491684us) ts_tablet_manager.cc:1145] Deleting tablet bfa48f94ac0e48e39578a6e8af5c8e9c
0629 01:51:50.387237 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.390821 (+ 3584us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.393621 (+ 2800us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.455235 (+ 61614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491472}
I20250629 01:51:50.455564 1157 tablet_replica.cc:331] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:50.455710 1157 raft_consensus.cc:2241] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.455821 1157 raft_consensus.cc:2270] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.456236 1157 ts_tablet_manager.cc:1905] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.456524 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc67c212516748ae99c1c8e51313aecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.457720 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet bfa48f94ac0e48e39578a6e8af5c8e9c (table default.Table_d7032fa31e084ecc8861910661af6259 [id=95ccd48727404118a0e2c26dd2103a12]) successfully deleted
I20250629 01:51:50.451695 1159 tablet_bootstrap.cc:492] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:50.459237 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4c7b76cb56b24604b13ee29c7f576ce2
I20250629 01:51:50.460270 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40266441c3684726a58a5029779395fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.461279 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.461458 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_629fa2d304cf42d2b686b6c46ab611fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.461900 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_629fa2d304cf42d2b686b6c46ab611fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.463235 877 tablet_service.cc:1515] Processing DeleteTablet for tablet b7c7f2ac44734a39ab6434a74c91b9e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.408488 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 44ab886d05e0468ab0e7b156a32dd392 with delete_type TABLET_DATA_DELETED (replaced by tablet f4609264b54241a0af744585f8bdbc89) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.467039 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b54228f0013424588a355ceb17d9784 with delete_type TABLET_DATA_DELETED (replaced by tablet ac560295fddd44368e6f9bc543f7cbfa) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.468943 1824 leader_election.cc:290] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.471030 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 725) took 1177 ms. Trace:
I20250629 01:51:50.471096 1158 rpcz_store.cc:276] 0629 01:51:49.293946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.312134 (+ 18188us) service_pool.cc:225] Handling call
0629 01:51:50.207339 (+895205us) ts_tablet_manager.cc:1145] Deleting tablet 0827ec622ff04ca6b2fec50e40b9e1de
0629 01:51:50.207351 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.437049 (+229698us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.439412 (+ 2363us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.471013 (+ 31601us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1105,"tablet-delete.queue_time_us":892876}
I20250629 01:51:50.471364 1158 tablet_replica.cc:331] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.471509 1158 raft_consensus.cc:2241] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.471542 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a90b206d58d04acdb1a8cb0c45d68c31" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.471632 1158 raft_consensus.cc:2270] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:50.472051 712 leader_election.cc:343] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:50.473678 710 leader_election.cc:343] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a90b206d58d04acdb1a8cb0c45d68c31
I20250629 01:51:50.473791 710 leader_election.cc:304] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:50.473971 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.474131 1824 raft_consensus.cc:2747] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.475858 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0827ec622ff04ca6b2fec50e40b9e1de (table default.Table_8c1fbbaabe004466858cce35b18194b1 [id=700ef08e094b44369154076e65d9ea94]) successfully deleted
I20250629 01:51:50.476270 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.477236 1158 ts_tablet_manager.cc:1905] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.483807 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_62002e6a41684d369e22fb34ba498d8f" } new_table_name: "default.Table_af561261ab3e4a8f933991231290368e" modify_external_catalogs: true
W20250629 01:51:50.484843 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60732c68906b4805bfa63cc4dafb4ceb consensus state has no leader
W20250629 01:51:50.389789 1893 consensus_queue.cc:1162] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250629 01:51:50.206928 755 tablet_service.cc:1468] Processing CreateTablet for tablet 9e0d550773c949df9992be1f9f01b548 (DEFAULT_TABLE table=default.Table_92114ac6ae69479e86909ca0e6744dec [id=17b2a91306024c68aa06426b022f6111]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.485181 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ef090d4a5e444517a98a92efc4d7f581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.485430 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0d550773c949df9992be1f9f01b548. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:50.485592 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef090d4a5e444517a98a92efc4d7f581 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.486604 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 86099eb9e40a4ea389afc059f90a97ef
I20250629 01:51:50.487125 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_55565d53ffdd403595e1e5cb1fd7effb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.487507 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55565d53ffdd403595e1e5cb1fd7effb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.489637 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ee987a9e3e4f19a99e07cb878b7511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.491180 1156 ts_tablet_manager.cc:1905] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.492209 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0eeb0ef02c416abc5665f058c6bd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.494127 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc67c212516748ae99c1c8e51313aecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.494668 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40266441c3684726a58a5029779395fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.494928 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf110bfaab6948f3a32cdf45241b31a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.495931 1156 ts_tablet_manager.cc:1918] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:50.496006 1156 log.cc:1199] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/26d85fd3d3dd4b07a812c019e1a5c748
I20250629 01:51:50.496345 1156 ts_tablet_manager.cc:1939] T 26d85fd3d3dd4b07a812c019e1a5c748 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.497084 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 109e19ed0ea64fa4985e93efbaca5107. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.497764 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 86099eb9e40a4ea389afc059f90a97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.497874 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 26d85fd3d3dd4b07a812c019e1a5c748 (table default.Table_331d3b611f4f46748132e37fb1a8e872 [id=b9df439e13104b14be7c1b522b04ec13]) successfully deleted
W20250629 01:51:50.498078 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9b54228f0013424588a355ceb17d9784 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b54228f0013424588a355ceb17d9784 already in progress: creating tablet
W20250629 01:51:50.498260 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 44ab886d05e0468ab0e7b156a32dd392 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ab886d05e0468ab0e7b156a32dd392 already in progress: creating tablet
W20250629 01:51:50.498461 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4c7b76cb56b24604b13ee29c7f576ce2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c7b76cb56b24604b13ee29c7f576ce2 already in progress: creating tablet
W20250629 01:51:50.498654 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 86099eb9e40a4ea389afc059f90a97ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86099eb9e40a4ea389afc059f90a97ef already in progress: creating tablet
I20250629 01:51:50.499873 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b54228f0013424588a355ceb17d9784 with delete_type TABLET_DATA_DELETED (replaced by tablet ac560295fddd44368e6f9bc543f7cbfa) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.501603 1155 tablet_bootstrap.cc:492] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.501722 1155 ts_tablet_manager.cc:1397] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.101s user 0.025s sys 0.011s
I20250629 01:51:50.502324 1155 raft_consensus.cc:357] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.502485 1155 raft_consensus.cc:383] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.502552 1155 raft_consensus.cc:738] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.502737 1850 raft_consensus.cc:513] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.502794 1155 consensus_queue.cc:260] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.503260 1850 leader_election.cc:290] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.503310 1155 ts_tablet_manager.cc:1428] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:50.503538 1155 tablet_bootstrap.cc:492] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.503857 1154 tablet_bootstrap.cc:492] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.503888 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:50.503957 1154 ts_tablet_manager.cc:1397] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.008s
W20250629 01:51:50.504470 712 leader_election.cc:343] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.504582 1154 raft_consensus.cc:357] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.504703 1154 raft_consensus.cc:383] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.504741 1154 raft_consensus.cc:738] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.504973 1155 tablet_bootstrap.cc:654] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.504913 1154 consensus_queue.cc:260] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.507052 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:50.507174 1048 raft_consensus.cc:3058] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:50.507920 1154 ts_tablet_manager.cc:1428] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250629 01:51:50.508860 1154 tablet_bootstrap.cc:492] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.514066 1159 tablet_bootstrap.cc:654] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.515777 1155 tablet_bootstrap.cc:492] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.515862 1155 ts_tablet_manager.cc:1397] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20250629 01:51:50.516448 1155 raft_consensus.cc:357] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.516575 1155 raft_consensus.cc:383] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.516619 1155 raft_consensus.cc:738] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.516789 1155 consensus_queue.cc:260] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.517553 1155 ts_tablet_manager.cc:1428] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:50.517751 1155 tablet_bootstrap.cc:492] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.519063 1155 tablet_bootstrap.cc:654] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.521968 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 86099eb9e40a4ea389afc059f90a97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.532495 1048 raft_consensus.cc:2466] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:50.532846 710 leader_election.cc:304] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:50.533300 1850 raft_consensus.cc:2802] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:50.533428 1850 raft_consensus.cc:695] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:50.533643 1850 consensus_queue.cc:237] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.535689 1154 tablet_bootstrap.cc:654] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.540259 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5a3752f8a275400da8cfff249f2dc6ee
W20250629 01:51:50.541882 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 621abed696cd4284bf8ee0223df312ee consensus state has no leader
I20250629 01:51:50.542220 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c5ad96e3c69d4a29ab6c662b82774b53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.542497 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a3752f8a275400da8cfff249f2dc6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:50.542718 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5ad96e3c69d4a29ab6c662b82774b53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.543316 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1591a004d955474d8456f3f4484e8d5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.543751 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1591a004d955474d8456f3f4484e8d5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.543946 449 master_service.cc:948] ReplaceTablet: received request to replace tablet e508393b9cb044c4b90194c8aab3184a from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.544207 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a3752f8a275400da8cfff249f2dc6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:50.546072 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5a3752f8a275400da8cfff249f2dc6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a3752f8a275400da8cfff249f2dc6ee already in progress: creating tablet
I20250629 01:51:50.550741 1154 tablet_bootstrap.cc:492] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.550890 1154 ts_tablet_manager.cc:1397] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.002s
I20250629 01:51:50.551493 1154 raft_consensus.cc:357] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.551687 1154 raft_consensus.cc:383] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.551764 1154 raft_consensus.cc:738] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.551965 1154 consensus_queue.cc:260] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.555114 1154 ts_tablet_manager.cc:1428] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250629 01:51:50.556751 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 86099eb9e40a4ea389afc059f90a97ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.556969 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a3752f8a275400da8cfff249f2dc6ee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.558687 1155 tablet_bootstrap.cc:492] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
W20250629 01:51:50.558734 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb98aa9d365541969ea1dc70960760b2 consensus state has no leader
I20250629 01:51:50.558864 1155 ts_tablet_manager.cc:1397] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.009s
I20250629 01:51:50.559443 1155 raft_consensus.cc:357] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.559645 1155 raft_consensus.cc:383] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.559738 1155 raft_consensus.cc:738] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.559984 1155 consensus_queue.cc:260] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.560299 1154 tablet_bootstrap.cc:492] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.560974 1155 ts_tablet_manager.cc:1428] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:50.561259 1155 tablet_bootstrap.cc:492] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.561546 1154 tablet_bootstrap.cc:654] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.562467 1155 tablet_bootstrap.cc:654] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.563001 1157 ts_tablet_manager.cc:1918] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.564867 1849 raft_consensus.cc:491] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.568344 1849 raft_consensus.cc:513] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.569010 1849 leader_election.cc:290] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:50.569985 850 leader_election.cc:343] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 75fd4ffcf896419b81a1c0eb7c5aeeec
W20250629 01:51:50.570629 850 leader_election.cc:343] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 75fd4ffcf896419b81a1c0eb7c5aeeec
I20250629 01:51:50.570736 850 leader_election.cc:304] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.571111 1849 raft_consensus.cc:2747] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.571314 1157 log.cc:1199] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8a4e5b66c75942cc8e3e9dfe5a69bf8f
I20250629 01:51:50.571689 1157 ts_tablet_manager.cc:1939] T 8a4e5b66c75942cc8e3e9dfe5a69bf8f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:50.575433 1158 ts_tablet_manager.cc:1918] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.577813 1155 tablet_bootstrap.cc:492] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.577749 1158 log.cc:1199] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e82b1352bb4d49769808dc402c85890f
I20250629 01:51:50.577967 1155 ts_tablet_manager.cc:1397] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20250629 01:51:50.578307 1158 ts_tablet_manager.cc:1939] T e82b1352bb4d49769808dc402c85890f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.578383 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 662) took 1646 ms. Trace:
I20250629 01:51:50.578501 1155 raft_consensus.cc:357] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.578482 1157 rpcz_store.cc:276] 0629 01:51:48.931624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.936604 (+ 4980us) service_pool.cc:225] Handling call
0629 01:51:50.455519 (+1518915us) ts_tablet_manager.cc:1145] Deleting tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f
0629 01:51:50.455528 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.553693 (+ 98165us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.562930 (+ 9237us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.578365 (+ 15435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518504}
I20250629 01:51:50.578697 1155 raft_consensus.cc:383] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.578760 1155 raft_consensus.cc:738] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.578956 1155 consensus_queue.cc:260] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.579494 1155 ts_tablet_manager.cc:1428] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:50.579546 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 723) took 1285 ms. Trace:
I20250629 01:51:50.579751 1155 tablet_bootstrap.cc:492] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.580354 1158 rpcz_store.cc:276] 0629 01:51:49.293791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.315866 (+ 22075us) service_pool.cc:225] Handling call
0629 01:51:50.471331 (+1155465us) ts_tablet_manager.cc:1145] Deleting tablet e82b1352bb4d49769808dc402c85890f
0629 01:51:50.471338 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.480846 (+ 9508us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.575351 (+ 94505us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.579530 (+ 4179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155281}
I20250629 01:51:50.580854 1155 tablet_bootstrap.cc:654] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.580940 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e82b1352bb4d49769808dc402c85890f (table default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]) successfully deleted
I20250629 01:51:50.581298 1158 tablet_replica.cc:331] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.581710 1158 raft_consensus.cc:2241] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.581909 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8a4e5b66c75942cc8e3e9dfe5a69bf8f (table default.Table_5f4d9b39ee1546eca5a1fb0701b2b93f [id=d5edb2619f57401898104c5d2d036b33]) successfully deleted
I20250629 01:51:50.582160 1158 raft_consensus.cc:2270] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.582803 1158 ts_tablet_manager.cc:1905] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.583894 1157 tablet_replica.cc:331] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:50.584084 1157 raft_consensus.cc:2241] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.584275 1157 raft_consensus.cc:2270] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.584803 1157 ts_tablet_manager.cc:1905] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.586860 1159 tablet_bootstrap.cc:492] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:50.586980 1159 ts_tablet_manager.cc:1397] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.135s user 0.001s sys 0.005s
I20250629 01:51:50.587795 1159 raft_consensus.cc:357] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.587970 1159 raft_consensus.cc:383] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.588012 1159 raft_consensus.cc:738] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.588419 1159 consensus_queue.cc:260] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.590199 1155 tablet_bootstrap.cc:492] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.590348 1155 ts_tablet_manager.cc:1397] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250629 01:51:50.590855 1155 raft_consensus.cc:357] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.591017 1155 raft_consensus.cc:383] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.591086 1155 raft_consensus.cc:738] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.591269 1155 consensus_queue.cc:260] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.591653 1155 ts_tablet_manager.cc:1428] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:50.591955 1155 tablet_bootstrap.cc:492] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.592907 1158 ts_tablet_manager.cc:1918] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.593050 1155 tablet_bootstrap.cc:654] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.594305 1158 log.cc:1199] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/69f0ae4c51924370896c624affde6761
I20250629 01:51:50.594696 1158 ts_tablet_manager.cc:1939] T 69f0ae4c51924370896c624affde6761 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.595598 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 724) took 1301 ms. Trace:
I20250629 01:51:50.596346 1158 rpcz_store.cc:276] 0629 01:51:49.293868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.316107 (+ 22239us) service_pool.cc:225] Handling call
0629 01:51:50.581252 (+1265145us) ts_tablet_manager.cc:1145] Deleting tablet 69f0ae4c51924370896c624affde6761
0629 01:51:50.581261 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.588894 (+ 7633us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.592831 (+ 3937us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.595580 (+ 2749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1265002}
I20250629 01:51:50.596571 1159 ts_tablet_manager.cc:1428] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:50.596956 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 69f0ae4c51924370896c624affde6761 (table default.Table_becfa32b77bb40b1b9df61126ffe628e [id=1c654e447e1b46f1a4570b41df8b1861]) successfully deleted
I20250629 01:51:50.597563 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 731) took 1244 ms. Trace:
I20250629 01:51:50.598152 1158 rpcz_store.cc:276] 0629 01:51:49.353360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.370489 (+ 17129us) service_pool.cc:225] Handling call
0629 01:51:50.597457 (+1226968us) ts_tablet_manager.cc:1145] Deleting tablet 0bef2747f0c04f498079906e40bac24c
0629 01:51:50.597466 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.597548 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226513}
W20250629 01:51:50.599861 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0bef2747f0c04f498079906e40bac24c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bef2747f0c04f498079906e40bac24c already in progress: creating tablet
I20250629 01:51:50.601003 1155 tablet_bootstrap.cc:492] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.601087 1155 ts_tablet_manager.cc:1397] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250629 01:51:50.601293 1159 tablet_bootstrap.cc:492] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:50.601547 1155 raft_consensus.cc:357] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.601660 1155 raft_consensus.cc:383] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.601694 1155 raft_consensus.cc:738] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.601823 1155 consensus_queue.cc:260] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.602246 1155 ts_tablet_manager.cc:1428] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:50.602473 1155 tablet_bootstrap.cc:492] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.602871 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e508393b9cb044c4b90194c8aab3184a
I20250629 01:51:50.603322 449 catalog_manager.cc:6652] ReplaceTablet: tablet e508393b9cb044c4b90194c8aab3184a deleted and replaced by tablet 86b826e303c14f5ca0af6a336673860b
I20250629 01:51:50.603482 1155 tablet_bootstrap.cc:654] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.603606 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet e508393b9cb044c4b90194c8aab3184a with delete_type TABLET_DATA_DELETED (replaced by tablet 86b826e303c14f5ca0af6a336673860b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.603943 738 tablet_service.cc:1515] Processing DeleteTablet for tablet e508393b9cb044c4b90194c8aab3184a with delete_type TABLET_DATA_DELETED (replaced by tablet 86b826e303c14f5ca0af6a336673860b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.604243 1156 tablet_replica.cc:331] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.604377 1156 raft_consensus.cc:2241] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.604524 1156 raft_consensus.cc:2270] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.604957 1156 ts_tablet_manager.cc:1905] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.609477 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b7a640451934bff8672cd88dc86c529 with delete_type TABLET_DATA_DELETED (replaced by tablet d456410cd09845dabd979c123f70741b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.609681 878 tablet_service.cc:1515] Processing DeleteTablet for tablet e508393b9cb044c4b90194c8aab3184a with delete_type TABLET_DATA_DELETED (replaced by tablet 86b826e303c14f5ca0af6a336673860b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.611748 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4e176f09e49f499982f892aaaeb1d681" } new_table_name: "default.Table_dacb5d3de26e4c82bc85838c8621390e" modify_external_catalogs: true
I20250629 01:51:50.613981 1156 ts_tablet_manager.cc:1918] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:50.614169 1156 log.cc:1199] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e508393b9cb044c4b90194c8aab3184a
I20250629 01:51:50.614720 1156 ts_tablet_manager.cc:1939] T e508393b9cb044c4b90194c8aab3184a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.618180 1159 tablet_bootstrap.cc:654] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.621309 1849 raft_consensus.cc:491] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.621416 1849 raft_consensus.cc:513] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.621942 1849 leader_election.cc:290] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:50.623328 850 leader_election.cc:343] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b7c7f2ac44734a39ab6434a74c91b9e4
W20250629 01:51:50.623499 850 leader_election.cc:343] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: b7c7f2ac44734a39ab6434a74c91b9e4
I20250629 01:51:50.623574 850 leader_election.cc:304] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.623921 1849 raft_consensus.cc:2747] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:50.624536 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01d95356c7a7419bb56ddf67d451eadc consensus state has no leader
I20250629 01:51:50.628468 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a60215417c34bedafbf12172eab469a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.628638 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.628827 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 with delete_type TABLET_DATA_DELETED (replaced by tablet bccdcd9de08648bb925db6fe56f97d20) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.629534 754 tablet_service.cc:1515] Processing DeleteTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.630249 738 tablet_service.cc:1468] Processing CreateTablet for tablet f4609264b54241a0af744585f8bdbc89 (DEFAULT_TABLE table=default.Table_d2e4a152a81f434eb4a24f2dd01af367 [id=35de30bc6f17483eb80fef1bf8ce6824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.630548 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4609264b54241a0af744585f8bdbc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.631410 454 catalog_manager.cc:5582] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:50.631752 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:50.633251 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2aca26af31ed4bc994f4d62603edc881
I20250629 01:51:50.633932 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 2aca26af31ed4bc994f4d62603edc881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.634277 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 2aca26af31ed4bc994f4d62603edc881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.634451 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 2aca26af31ed4bc994f4d62603edc881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.637045 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet ea00b4559e03410885180286b5e9fd10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.637291 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.638357 1157 ts_tablet_manager.cc:1918] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:50.638442 1157 log.cc:1199] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/82d58c88a6994d828b444c5814b287ff
I20250629 01:51:50.638720 1157 ts_tablet_manager.cc:1939] T 82d58c88a6994d828b444c5814b287ff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:50.638942 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 661) took 1712 ms. Trace:
I20250629 01:51:50.639003 1157 rpcz_store.cc:276] 0629 01:51:48.926893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:48.941550 (+ 14657us) service_pool.cc:225] Handling call
0629 01:51:50.583850 (+1642300us) ts_tablet_manager.cc:1145] Deleting tablet 82d58c88a6994d828b444c5814b287ff
0629 01:51:50.583856 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.634514 (+ 50658us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.638282 (+ 3768us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.638931 (+ 649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642104}
I20250629 01:51:50.639351 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 82d58c88a6994d828b444c5814b287ff (table default.Table_1961bae4492d41e8b6c8536a9f324977 [id=74626a8a46524de4b96a5e946526aefe]) successfully deleted
I20250629 01:51:50.640476 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]) successfully deleted
I20250629 01:51:50.641371 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 39d2adf801934179825f96a5f7b3c303
I20250629 01:51:50.642187 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 39d2adf801934179825f96a5f7b3c303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.642216 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 39d2adf801934179825f96a5f7b3c303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.643110 1156 tablet_replica.cc:331] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.643297 1156 raft_consensus.cc:2241] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.643455 1156 raft_consensus.cc:2270] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.644109 1156 ts_tablet_manager.cc:1905] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.646154 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 39d2adf801934179825f96a5f7b3c303 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.646360 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f4609264b54241a0af744585f8bdbc89 (DEFAULT_TABLE table=default.Table_d2e4a152a81f434eb4a24f2dd01af367 [id=35de30bc6f17483eb80fef1bf8ce6824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.646483 885 tablet_service.cc:1468] Processing CreateTablet for tablet f4609264b54241a0af744585f8bdbc89 (DEFAULT_TABLE table=default.Table_d2e4a152a81f434eb4a24f2dd01af367 [id=35de30bc6f17483eb80fef1bf8ce6824]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.646764 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4609264b54241a0af744585f8bdbc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.646782 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4609264b54241a0af744585f8bdbc89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.649104 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.649819 746 tablet_service.cc:1515] Processing DeleteTablet for tablet dd709b557b334434a6c2c4a60a996342 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.649996 755 tablet_service.cc:1515] Processing DeleteTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.650141 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a574f95ac2a4193a563e63cbe120768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.650396 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d57756db239b4178a430514058de8274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.650926 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d57756db239b4178a430514058de8274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.652343 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71248d2aeaee415194b2450bdb8def2e" } new_table_name: "default.Table_fd2fc7860a3040fe934229df61786a2a" modify_external_catalogs: true
I20250629 01:51:50.657646 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_bb9d3aa0a3084c80a33203f45b4978eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.658085 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb9d3aa0a3084c80a33203f45b4978eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.660352 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_93f95da30d034c31b5c7f2c52229fcf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.660898 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93f95da30d034c31b5c7f2c52229fcf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.661916 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dacb5d3de26e4c82bc85838c8621390e" } modify_external_catalogs: true
I20250629 01:51:50.664057 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 732) took 1258 ms. Trace:
I20250629 01:51:50.664176 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dacb5d3de26e4c82bc85838c8621390e" } modify_external_catalogs: true
I20250629 01:51:50.664193 1158 rpcz_store.cc:276] 0629 01:51:49.405803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.405928 (+ 125us) service_pool.cc:225] Handling call
0629 01:51:50.663947 (+1258019us) ts_tablet_manager.cc:1145] Deleting tablet 22567416bfde4e2b843cf5c7a214ec6c
0629 01:51:50.663955 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.664038 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257705}
W20250629 01:51:50.664598 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 22567416bfde4e2b843cf5c7a214ec6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22567416bfde4e2b843cf5c7a214ec6c already in progress: creating tablet
I20250629 01:51:50.664891 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 734) took 1232 ms. Trace:
I20250629 01:51:50.664952 1158 rpcz_store.cc:276] 0629 01:51:49.432381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.442926 (+ 10545us) service_pool.cc:225] Handling call
0629 01:51:50.664799 (+1221873us) ts_tablet_manager.cc:1145] Deleting tablet 4aa8dfd712c043ad9f8ff56b064bb8fd
0629 01:51:50.664807 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.664876 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1221711}
W20250629 01:51:50.665342 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa8dfd712c043ad9f8ff56b064bb8fd already in progress: creating tablet
I20250629 01:51:50.665640 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 733) took 1244 ms. Trace:
I20250629 01:51:50.665705 1158 rpcz_store.cc:276] 0629 01:51:49.421312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.443557 (+ 22245us) service_pool.cc:225] Handling call
0629 01:51:50.665556 (+1221999us) ts_tablet_manager.cc:1145] Deleting tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b
0629 01:51:50.665563 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.665627 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1221846}
W20250629 01:51:50.666023 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b already in progress: creating tablet
I20250629 01:51:50.667829 1154 tablet_bootstrap.cc:492] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.667976 1154 ts_tablet_manager.cc:1397] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.108s user 0.015s sys 0.002s
I20250629 01:51:50.668651 1154 raft_consensus.cc:357] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.668843 1154 raft_consensus.cc:383] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.668933 1154 raft_consensus.cc:738] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.669165 1154 consensus_queue.cc:260] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.673185 1850 raft_consensus.cc:491] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.673317 1850 raft_consensus.cc:513] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.673826 1850 leader_election.cc:290] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.674263 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69980d5bf37d4001a6196b22c34ccc94" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:50.674628 712 leader_election.cc:343] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:50.674916 710 leader_election.cc:343] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:50.675001 710 leader_election.cc:304] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:50.675307 1850 raft_consensus.cc:2747] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:50.679538 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d02d437359a643b59d1bd41dfe77a0fc consensus state has no leader
I20250629 01:51:50.682549 1154 ts_tablet_manager.cc:1428] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250629 01:51:50.683138 1159 tablet_bootstrap.cc:492] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:50.683311 1159 ts_tablet_manager.cc:1397] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20250629 01:51:50.685984 1154 tablet_bootstrap.cc:492] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.685920 1159 raft_consensus.cc:357] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.686053 1159 raft_consensus.cc:383] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.686105 1159 raft_consensus.cc:738] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.686297 1159 consensus_queue.cc:260] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.687386 1154 tablet_bootstrap.cc:654] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.687992 1158 tablet_replica.cc:331] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.688643 1158 raft_consensus.cc:2241] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.688839 1158 raft_consensus.cc:2270] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.689079 1156 ts_tablet_manager.cc:1918] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.689560 1158 ts_tablet_manager.cc:1905] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.689690 1156 log.cc:1199] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8a60215417c34bedafbf12172eab469a
I20250629 01:51:50.690392 1156 ts_tablet_manager.cc:1939] T 8a60215417c34bedafbf12172eab469a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.693606 1159 ts_tablet_manager.cc:1428] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250629 01:51:50.694464 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8a60215417c34bedafbf12172eab469a (table default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]) successfully deleted
I20250629 01:51:50.694886 1156 tablet_replica.cc:331] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.695102 1156 raft_consensus.cc:2241] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.695700 1156 raft_consensus.cc:2270] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.695987 1155 tablet_bootstrap.cc:492] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.696171 1155 ts_tablet_manager.cc:1397] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.094s user 0.023s sys 0.023s
I20250629 01:51:50.696460 1156 ts_tablet_manager.cc:1905] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.697134 1159 tablet_bootstrap.cc:492] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:50.696784 1155 raft_consensus.cc:357] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.697961 1155 raft_consensus.cc:383] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.698055 1155 raft_consensus.cc:738] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.698305 1155 consensus_queue.cc:260] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.698730 1159 tablet_bootstrap.cc:654] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.698844 1155 ts_tablet_manager.cc:1428] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:50.699115 1155 tablet_bootstrap.cc:492] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.699684 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3827f711f3174f66b469302de8091583" } modify_external_catalogs: true
I20250629 01:51:50.699842 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3827f711f3174f66b469302de8091583" } modify_external_catalogs: true
I20250629 01:51:50.701145 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2ebed88287e94163bf75e1ee852ecc4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.701620 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ebed88287e94163bf75e1ee852ecc4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.702546 1154 tablet_bootstrap.cc:492] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.702693 1154 ts_tablet_manager.cc:1397] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20250629 01:51:50.702780 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5aaf33ae5de2462c9d6aadf0a0c9e851" } new_table_name: "default.Table_23786b49be3a423389f38c092b3bbbf8" modify_external_catalogs: true
I20250629 01:51:50.703271 1154 raft_consensus.cc:357] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.703390 1154 raft_consensus.cc:383] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.703431 1154 raft_consensus.cc:738] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.703653 1154 consensus_queue.cc:260] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.706072 1154 ts_tablet_manager.cc:1428] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:50.708735 1154 tablet_bootstrap.cc:492] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.711896 1154 tablet_bootstrap.cc:654] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.713451 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 01d95356c7a7419bb56ddf67d451eadc
I20250629 01:51:50.714648 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 01d95356c7a7419bb56ddf67d451eadc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.715991 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:50.716313 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 01d95356c7a7419bb56ddf67d451eadc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.716646 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.719223 1155 tablet_bootstrap.cc:654] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.719260 1710 raft_consensus.cc:491] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.719362 1710 raft_consensus.cc:513] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.719869 1710 leader_election.cc:290] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:50.720152 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b680859a1a4c56afe33f5b1b3c0793" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:50.716702 1158 ts_tablet_manager.cc:1918] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.720410 1158 log.cc:1199] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8a60215417c34bedafbf12172eab469a
I20250629 01:51:50.720664 1158 ts_tablet_manager.cc:1939] T 8a60215417c34bedafbf12172eab469a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:50.720876 986 leader_election.cc:343] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.710323 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d61687e19d994e87b09e55bd3b9476d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.714598 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 01d95356c7a7419bb56ddf67d451eadc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.720463 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b680859a1a4c56afe33f5b1b3c0793" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.721438 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8a60215417c34bedafbf12172eab469a (table default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]) successfully deleted
W20250629 01:51:50.721766 988 leader_election.cc:343] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.721854 988 leader_election.cc:304] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.722152 1710 raft_consensus.cc:2747] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.723577 1155 tablet_bootstrap.cc:492] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.723673 1155 ts_tablet_manager.cc:1397] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250629 01:51:50.724363 1155 raft_consensus.cc:357] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.724495 1155 raft_consensus.cc:383] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.724541 1155 raft_consensus.cc:738] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.724695 1155 consensus_queue.cc:260] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.725097 1155 ts_tablet_manager.cc:1428] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:50.725423 1155 tablet_bootstrap.cc:492] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.726845 1155 tablet_bootstrap.cc:654] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.732410 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 77eae224763d464496066b4a98efe5ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:50.735394 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 97d0fbb691514d51a1ca71f32a666c2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d0fbb691514d51a1ca71f32a666c2d already in progress: creating tablet
I20250629 01:51:50.737850 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2d37334db2b9465a8fcf5e849d5d3b0c
I20250629 01:51:50.738587 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d37334db2b9465a8fcf5e849d5d3b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.739228 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d37334db2b9465a8fcf5e849d5d3b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.739256 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d37334db2b9465a8fcf5e849d5d3b0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.742101 1157 tablet_replica.cc:331] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:50.742378 1157 raft_consensus.cc:2241] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.742570 1157 raft_consensus.cc:2270] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.742784 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f7db52cd26e34c54b5fa3ba338525f24" } modify_external_catalogs: true
I20250629 01:51:50.742945 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f7db52cd26e34c54b5fa3ba338525f24" } modify_external_catalogs: true
I20250629 01:51:50.743667 1158 tablet_replica.cc:331] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:50.744828 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 02bd46ceac1e4766ada4c4d2e95fda00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02bd46ceac1e4766ada4c4d2e95fda00 already in progress: creating tablet
W20250629 01:51:50.745065 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 43b845c3e7174618bd5c394d7206a634 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b845c3e7174618bd5c394d7206a634 already in progress: creating tablet
I20250629 01:51:50.743824 1158 raft_consensus.cc:2241] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:50.745263 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet dc841eb171d04b118b1e705ceed4c861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc841eb171d04b118b1e705ceed4c861 already in progress: creating tablet
I20250629 01:51:50.745319 1158 raft_consensus.cc:2270] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.747246 1158 ts_tablet_manager.cc:1905] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.748800 1157 ts_tablet_manager.cc:1905] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.749339 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d61ce7701ae64cfd903fdcdbc989bc74" } new_table_name: "default.Table_5b205ff41f0748eaa0dcd3aed3820e51" modify_external_catalogs: true
I20250629 01:51:50.749398 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.749851 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.752331 448 master_service.cc:948] ReplaceTablet: received request to replace tablet e3b94f8250a143b096703f2b0386e3cd from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.754319 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e67aa47b96b74730985e975276f3745b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.754810 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e67aa47b96b74730985e975276f3745b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:50.757337 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4234c00081e847e9ba6bb70e630ba764 consensus state has no leader
I20250629 01:51:50.759922 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3ec720103be44b36a6f7d15307a437ff" } new_table_name: "default.Table_52330030fd4140c1af870dcfcb4dc269" modify_external_catalogs: true
I20250629 01:51:50.763139 1156 ts_tablet_manager.cc:1918] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.763325 1156 log.cc:1199] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4ed143377709465d9ae5df7c2ec3ca47
I20250629 01:51:50.763731 1156 ts_tablet_manager.cc:1939] T 4ed143377709465d9ae5df7c2ec3ca47 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.764312 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f52937830a924d6d8338d78da24b84cd
I20250629 01:51:50.765437 1158 ts_tablet_manager.cc:1918] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.765789 1158 log.cc:1199] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0b16b6840c24484b90218fffc1da9621
I20250629 01:51:50.766000 743 tablet_service.cc:1515] Processing DeleteTablet for tablet f52937830a924d6d8338d78da24b84cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.765997 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet f52937830a924d6d8338d78da24b84cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:50.766362 1158 ts_tablet_manager.cc:1939] T 0b16b6840c24484b90218fffc1da9621 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.766487 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f52937830a924d6d8338d78da24b84cd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.767735 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0b16b6840c24484b90218fffc1da9621 (table default.Table_10a0ae1fcff049dda2c11881c3191fa3 [id=14974743edd741fbab51f9e8a4add096]) successfully deleted
W20250629 01:51:50.769773 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0ba9eabd06174b87a1485f4921895c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ba9eabd06174b87a1485f4921895c5f already in progress: creating tablet
W20250629 01:51:50.770581 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 40592f11f430469f9c8e82eba25b38ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40592f11f430469f9c8e82eba25b38ed already in progress: creating tablet
W20250629 01:51:50.771447 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 51b8e24f87b34f15b45270d4693dc7b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b8e24f87b34f15b45270d4693dc7b5 already in progress: creating tablet
W20250629 01:51:50.772694 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 476a6cf28db04f67bc23ddfc5cacad87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476a6cf28db04f67bc23ddfc5cacad87 already in progress: creating tablet
I20250629 01:51:50.772771 1158 tablet_replica.cc:331] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:50.772902 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c40a07f50971493cb353ed94e850719a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40a07f50971493cb353ed94e850719a already in progress: creating tablet
I20250629 01:51:50.772917 1158 raft_consensus.cc:2241] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.773028 1158 raft_consensus.cc:2270] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.777029 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a7f78eef06dd413cad0a63d1923e1263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.777463 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7f78eef06dd413cad0a63d1923e1263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.778439 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_da43c1c1f1c948849cef128c6cbaccfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.778841 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da43c1c1f1c948849cef128c6cbaccfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.779980 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1f13038d42254b24858c6b4d8af9d70a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.780483 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f13038d42254b24858c6b4d8af9d70a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.786593 1157 ts_tablet_manager.cc:1918] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.786690 1157 log.cc:1199] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d663e7dc7a384bf6bd0cbfa9d9e5e9f0
I20250629 01:51:50.786929 1157 ts_tablet_manager.cc:1939] T d663e7dc7a384bf6bd0cbfa9d9e5e9f0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:50.787168 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 663) took 1773 ms. Trace:
I20250629 01:51:50.787251 1157 rpcz_store.cc:276] 0629 01:51:49.013487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.013682 (+ 195us) service_pool.cc:225] Handling call
0629 01:51:50.742059 (+1728377us) ts_tablet_manager.cc:1145] Deleting tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0
0629 01:51:50.742068 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.783932 (+ 41864us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.786523 (+ 2591us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:50.787156 (+ 633us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1728237}
I20250629 01:51:50.787608 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 666) took 1746 ms. Trace:
I20250629 01:51:50.787642 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d663e7dc7a384bf6bd0cbfa9d9e5e9f0 (table default.Table_737eb1cf8b184a14b253c25d114a14b7 [id=eda52aa556274b26af9896e638b51e7a]) successfully deleted
I20250629 01:51:50.787675 1157 rpcz_store.cc:276] 0629 01:51:49.041280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.041321 (+ 41us) service_pool.cc:225] Handling call
0629 01:51:50.787520 (+1746199us) ts_tablet_manager.cc:1145] Deleting tablet 17acb9bcd7aa4a508d1695474a1b56df
0629 01:51:50.787528 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.787596 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746066}
I20250629 01:51:50.787945 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 668) took 1739 ms. Trace:
W20250629 01:51:50.788023 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 17acb9bcd7aa4a508d1695474a1b56df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17acb9bcd7aa4a508d1695474a1b56df already in progress: creating tablet
I20250629 01:51:50.789319 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4ed143377709465d9ae5df7c2ec3ca47 (table default.Table_ace97daf931b45ac9676c39144d2931f [id=1669d2f9d2b842bfb8fdbc03b965669a]) successfully deleted
I20250629 01:51:50.789645 1156 tablet_replica.cc:331] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.789914 1849 raft_consensus.cc:491] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.790046 1849 raft_consensus.cc:513] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.790356 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 543242e4a5064268ae1e5519b569f682 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.790654 1849 leader_election.cc:290] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.790933 1156 raft_consensus.cc:2241] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.791119 1156 raft_consensus.cc:2270] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:50.791318 850 leader_election.cc:343] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0b7a640451934bff8672cd88dc86c529
W20250629 01:51:50.791605 850 leader_election.cc:343] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0b7a640451934bff8672cd88dc86c529
I20250629 01:51:50.791692 850 leader_election.cc:304] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.791725 1156 ts_tablet_manager.cc:1905] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.791986 1849 raft_consensus.cc:2747] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.792811 1155 tablet_bootstrap.cc:492] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.792937 1155 ts_tablet_manager.cc:1397] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.005s
I20250629 01:51:50.793550 1155 raft_consensus.cc:357] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.793694 1155 raft_consensus.cc:383] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.793749 1155 raft_consensus.cc:738] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
W20250629 01:51:50.793937 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9dad47c452ca45039f44d08c43a7cca3 consensus state has no leader
I20250629 01:51:50.795979 1159 tablet_bootstrap.cc:492] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:50.796145 1159 ts_tablet_manager.cc:1397] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.001s
I20250629 01:51:50.796829 1155 consensus_queue.cc:260] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.797093 1159 raft_consensus.cc:357] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.797271 1159 raft_consensus.cc:383] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.797354 1159 raft_consensus.cc:738] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.797389 1158 ts_tablet_manager.cc:1905] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.797574 1159 consensus_queue.cc:260] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.797885 1155 ts_tablet_manager.cc:1428] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:50.800406 1849 raft_consensus.cc:491] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.800496 1849 raft_consensus.cc:513] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.796703 1154 tablet_bootstrap.cc:492] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.800747 1154 ts_tablet_manager.cc:1397] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.013s
I20250629 01:51:50.800940 1849 leader_election.cc:290] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.801343 1154 raft_consensus.cc:357] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:50.801440 850 leader_election.cc:343] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 4dbe70d4e4e74c6ba7db2766bd78c8ad
I20250629 01:51:50.801474 1154 raft_consensus.cc:383] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.801538 1154 raft_consensus.cc:738] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.801709 1849 raft_consensus.cc:491] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.801802 1849 raft_consensus.cc:513] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.801910 1155 tablet_bootstrap.cc:492] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.802238 1849 leader_election.cc:290] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:50.802772 850 leader_election.cc:343] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 4dbe70d4e4e74c6ba7db2766bd78c8ad
I20250629 01:51:50.802863 850 leader_election.cc:304] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.803419 1159 ts_tablet_manager.cc:1428] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250629 01:51:50.805182 1159 tablet_bootstrap.cc:492] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:50.806340 1159 tablet_bootstrap.cc:654] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.806900 1154 consensus_queue.cc:260] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.809590 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.810499 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_14492ecf17b84f3f8b5fb91233a242dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.810907 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14492ecf17b84f3f8b5fb91233a242dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.811717 1154 ts_tablet_manager.cc:1428] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250629 01:51:50.813694 1154 tablet_bootstrap.cc:492] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.809091 1849 raft_consensus.cc:2747] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.814793 1154 tablet_bootstrap.cc:654] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.815608 1156 ts_tablet_manager.cc:1918] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.819252 1158 ts_tablet_manager.cc:1918] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.823719 1156 log.cc:1199] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a90b206d58d04acdb1a8cb0c45d68c31
I20250629 01:51:50.824028 1156 ts_tablet_manager.cc:1939] T a90b206d58d04acdb1a8cb0c45d68c31 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.816535 1155 tablet_bootstrap.cc:654] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.826032 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ecc7812932cb496d8e92eee3befdce44" } new_table_name: "default.Table_62e935463ed04d11b4774769be58c37d" modify_external_catalogs: true
W20250629 01:51:50.827702 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce7a72db953646038a9d82dde04d76e4 consensus state has no leader
I20250629 01:51:50.829322 1158 log.cc:1199] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/26d85fd3d3dd4b07a812c019e1a5c748
I20250629 01:51:50.829608 1158 ts_tablet_manager.cc:1939] T 26d85fd3d3dd4b07a812c019e1a5c748 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.830376 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 26d85fd3d3dd4b07a812c019e1a5c748 (table default.Table_331d3b611f4f46748132e37fb1a8e872 [id=b9df439e13104b14be7c1b522b04ec13]) successfully deleted
I20250629 01:51:50.830793 1158 tablet_replica.cc:331] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.830986 1158 raft_consensus.cc:2241] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.831472 1710 raft_consensus.cc:491] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.831606 1710 raft_consensus.cc:513] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.831913 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 845f2422d6494c95a4738f3062c216c7 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:50.832067 1710 leader_election.cc:290] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:50.832844 1156 tablet_replica.cc:331] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.832924 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "798516e55ca747fcb2942026adb78af0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:50.832958 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a90b206d58d04acdb1a8cb0c45d68c31 (table default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]) successfully deleted
I20250629 01:51:50.833019 1154 tablet_bootstrap.cc:492] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.833037 1156 raft_consensus.cc:2241] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.833104 1154 ts_tablet_manager.cc:1397] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20250629 01:51:50.833158 1156 raft_consensus.cc:2270] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:50.833328 986 leader_election.cc:343] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.833626 1156 ts_tablet_manager.cc:1905] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.833695 1154 raft_consensus.cc:357] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.833832 1154 raft_consensus.cc:383] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.833889 1154 raft_consensus.cc:738] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.834148 1154 consensus_queue.cc:260] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.837662 1154 ts_tablet_manager.cc:1428] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:50.840855 1154 tablet_bootstrap.cc:492] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.842129 1154 tablet_bootstrap.cc:654] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.842784 1158 raft_consensus.cc:2270] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.843219 1158 ts_tablet_manager.cc:1905] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.844892 1155 tablet_bootstrap.cc:492] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.845012 1155 ts_tablet_manager.cc:1397] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.003s
I20250629 01:51:50.845731 1155 raft_consensus.cc:357] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.845889 1155 raft_consensus.cc:383] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.845945 1155 raft_consensus.cc:738] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.846122 1155 consensus_queue.cc:260] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.847682 1155 ts_tablet_manager.cc:1428] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:50.847901 1155 tablet_bootstrap.cc:492] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.849426 1155 tablet_bootstrap.cc:654] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.848948 1158 ts_tablet_manager.cc:1918] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.850878 1158 log.cc:1199] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b9b680859a1a4c56afe33f5b1b3c0793
I20250629 01:51:50.851225 1158 ts_tablet_manager.cc:1939] T b9b680859a1a4c56afe33f5b1b3c0793 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:50.852125 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7253ebd64bc14d1c89a2602b5f64e13a" } modify_external_catalogs: true
I20250629 01:51:50.852403 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b9b680859a1a4c56afe33f5b1b3c0793 (table default.Table_30467a20b5b54ca7894c405b22705bc2 [id=66af0a365c3f40deac642473f9125b10]) successfully deleted
I20250629 01:51:50.852437 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7253ebd64bc14d1c89a2602b5f64e13a" } modify_external_catalogs: true
W20250629 01:51:50.853678 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 44ab886d05e0468ab0e7b156a32dd392 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ab886d05e0468ab0e7b156a32dd392 already in progress: creating tablet
I20250629 01:51:50.853933 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d9cb1a00d22843f69edc93ba305abaad" } modify_external_catalogs: true
I20250629 01:51:50.854106 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d9cb1a00d22843f69edc93ba305abaad" } modify_external_catalogs: true
I20250629 01:51:50.854725 1048 raft_consensus.cc:1273] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:50.854844 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4c7b76cb56b24604b13ee29c7f576ce2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c7b76cb56b24604b13ee29c7f576ce2 already in progress: creating tablet
I20250629 01:51:50.855290 1861 consensus_queue.cc:1035] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:51:50.855458 712 consensus_peers.cc:489] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:51:50.856122 850 leader_election.cc:343] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: d61687e19d994e87b09e55bd3b9476d7
W20250629 01:51:50.856289 850 leader_election.cc:343] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d61687e19d994e87b09e55bd3b9476d7
I20250629 01:51:50.856366 850 leader_election.cc:304] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:50.856727 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9b54228f0013424588a355ceb17d9784 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b54228f0013424588a355ceb17d9784 already in progress: creating tablet
I20250629 01:51:50.856897 1915 raft_consensus.cc:2747] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.857323 1156 ts_tablet_manager.cc:1918] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.857458 1156 log.cc:1199] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:50.857753 1156 ts_tablet_manager.cc:1939] T d91fa0f049904a1fb5cbd38d1fd1693a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.857793 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "798516e55ca747fcb2942026adb78af0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.858232 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c712d978a3848b9bb3abdbce8314dec with delete_type TABLET_DATA_DELETED (replaced by tablet 0323947b135a4308903b1eed1ca4db71) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:50.859284 988 leader_election.cc:343] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.859367 988 leader_election.cc:304] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:50.859843 1710 raft_consensus.cc:2747] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:50.860275 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d91fa0f049904a1fb5cbd38d1fd1693a (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]) successfully deleted
W20250629 01:51:50.860956 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2aca26af31ed4bc994f4d62603edc881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aca26af31ed4bc994f4d62603edc881 already in progress: creating tablet
W20250629 01:51:50.861615 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 39d2adf801934179825f96a5f7b3c303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39d2adf801934179825f96a5f7b3c303 already in progress: creating tablet
I20250629 01:51:50.864112 1156 tablet_replica.cc:331] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.864312 1156 raft_consensus.cc:2241] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.864485 1156 raft_consensus.cc:2270] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.864845 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.865139 1156 ts_tablet_manager.cc:1905] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.866755 1154 tablet_bootstrap.cc:492] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.866887 1154 ts_tablet_manager.cc:1397] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.003s
I20250629 01:51:50.867725 1154 raft_consensus.cc:357] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.867906 1154 raft_consensus.cc:383] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.867969 1154 raft_consensus.cc:738] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.868747 1155 tablet_bootstrap.cc:492] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.868861 1155 ts_tablet_manager.cc:1397] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250629 01:51:50.869530 1155 raft_consensus.cc:357] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.869657 1155 raft_consensus.cc:383] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.869704 1155 raft_consensus.cc:738] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.869900 1155 consensus_queue.cc:260] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.870239 1154 consensus_queue.cc:260] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.871258 1154 ts_tablet_manager.cc:1428] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:50.871508 1154 tablet_bootstrap.cc:492] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:50.872988 1155 ts_tablet_manager.cc:1428] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250629 01:51:50.874199 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 86099eb9e40a4ea389afc059f90a97ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86099eb9e40a4ea389afc059f90a97ef already in progress: creating tablet
W20250629 01:51:50.874394 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5a3752f8a275400da8cfff249f2dc6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a3752f8a275400da8cfff249f2dc6ee already in progress: creating tablet
I20250629 01:51:50.874465 1155 tablet_bootstrap.cc:492] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.874604 1158 tablet_replica.cc:331] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:50.874866 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_46cfbca358ce43a99a1a1dc43600e2b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:50.875622 1158 raft_consensus.cc:2241] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
W20250629 01:51:50.875622 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46cfbca358ce43a99a1a1dc43600e2b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.875818 1155 tablet_bootstrap.cc:654] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.876204 1158 raft_consensus.cc:2270] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.876994 1158 ts_tablet_manager.cc:1905] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.877787 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e82b1352bb4d49769808dc402c85890f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.877352 1154 tablet_bootstrap.cc:654] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.880697 1156 ts_tablet_manager.cc:1918] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:50.881851 440 catalog_manager.cc:5001] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): alter failed for tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:50.882047 1156 log.cc:1199] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:50.882627 1156 ts_tablet_manager.cc:1939] T 69980d5bf37d4001a6196b22c34ccc94 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.883486 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 69980d5bf37d4001a6196b22c34ccc94 (table default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 [id=fd2351d583834130bb8213424bf96abb]) successfully deleted
I20250629 01:51:50.884217 1156 tablet_replica.cc:331] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.884506 1156 raft_consensus.cc:2241] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.884791 1156 raft_consensus.cc:2270] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.885484 1156 ts_tablet_manager.cc:1905] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.886916 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5407d7aa15024faab9edbf55a1485511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.887732 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5407d7aa15024faab9edbf55a1485511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.889709 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d2e4a152a81f434eb4a24f2dd01af367" } new_table_name: "default.Table_abe61aad409e43b68e638c756f3aad55" modify_external_catalogs: true
I20250629 01:51:50.788003 1157 rpcz_store.cc:276] 0629 01:51:49.048233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.048378 (+ 145us) service_pool.cc:225] Handling call
0629 01:51:50.787870 (+1739492us) ts_tablet_manager.cc:1145] Deleting tablet 52be184d0480495a894442d6f72a0f12
0629 01:51:50.787877 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.787933 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739370}
W20250629 01:51:50.890405 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 52be184d0480495a894442d6f72a0f12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52be184d0480495a894442d6f72a0f12 already in progress: creating tablet
I20250629 01:51:50.900966 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]). Expected version 3 got 0
I20250629 01:51:50.901121 1154 tablet_bootstrap.cc:492] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:50.901219 1154 ts_tablet_manager.cc:1397] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250629 01:51:50.902426 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:50.904884 885 tablet_service.cc:1468] Processing CreateTablet for tablet 989b26c3e2294005a1ebb4316c81ac29 (DEFAULT_TABLE table=default.Table_1591a004d955474d8456f3f4484e8d5f [id=9b59997abe6d438a8722061717ca2276]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.905277 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bc3d470df0d4814a655bcff68c956e2 consensus state has no leader
I20250629 01:51:50.905393 878 tablet_service.cc:1468] Processing CreateTablet for tablet 5c534f4fb7354fceb5f50df83292371a (DEFAULT_TABLE table=default.Table_ef090d4a5e444517a98a92efc4d7f581 [id=2518e68e6cf9435f8d8510cabd561ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.905848 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2521a476643c41c1a2e5282755a58091 (DEFAULT_TABLE table=default.Table_c5ad96e3c69d4a29ab6c662b82774b53 [id=35205188f6ee434a822596e1f23a2253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.906112 1928 tablet.cc:1722] T 8976cca816b442cc9a7cb4f23cc13016 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:50.907152 743 tablet_service.cc:1468] Processing CreateTablet for tablet 989b26c3e2294005a1ebb4316c81ac29 (DEFAULT_TABLE table=default.Table_1591a004d955474d8456f3f4484e8d5f [id=9b59997abe6d438a8722061717ca2276]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.907152 889 tablet_service.cc:1468] Processing CreateTablet for tablet f5e17d5171174efd9b628a0cc38b4f49 (DEFAULT_TABLE table=default.Table_629fa2d304cf42d2b686b6c46ab611fd [id=bf47771ababf4cc391d85a9c383690a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.907452 746 tablet_service.cc:1468] Processing CreateTablet for tablet 90eef69522e54b12b849d94e56355921 (DEFAULT_TABLE table=default.Table_b92827e379f148d4bd76b3e578adc695 [id=6faf9bc8cce84174a48928674dc25702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.907469 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989b26c3e2294005a1ebb4316c81ac29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.907773 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90eef69522e54b12b849d94e56355921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.894837 1155 tablet_bootstrap.cc:492] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.907971 1155 ts_tablet_manager.cc:1397] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20250629 01:51:50.908061 738 tablet_service.cc:1468] Processing CreateTablet for tablet 5c534f4fb7354fceb5f50df83292371a (DEFAULT_TABLE table=default.Table_ef090d4a5e444517a98a92efc4d7f581 [id=2518e68e6cf9435f8d8510cabd561ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.908367 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c534f4fb7354fceb5f50df83292371a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.908573 1155 raft_consensus.cc:357] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.908713 1155 raft_consensus.cc:383] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.908753 1155 raft_consensus.cc:738] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.908898 1155 consensus_queue.cc:260] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.909250 1155 ts_tablet_manager.cc:1428] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:50.891402 1927 tablet.cc:1722] T 8976cca816b442cc9a7cb4f23cc13016 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:50.911017 1155 tablet_bootstrap.cc:492] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.912294 754 tablet_service.cc:1468] Processing CreateTablet for tablet 2521a476643c41c1a2e5282755a58091 (DEFAULT_TABLE table=default.Table_c5ad96e3c69d4a29ab6c662b82774b53 [id=35205188f6ee434a822596e1f23a2253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.912525 1824 raft_consensus.cc:491] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:50.912588 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2521a476643c41c1a2e5282755a58091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.912611 1824 raft_consensus.cc:513] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.913015 1824 leader_election.cc:290] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.916108 755 tablet_service.cc:1468] Processing CreateTablet for tablet f5e17d5171174efd9b628a0cc38b4f49 (DEFAULT_TABLE table=default.Table_629fa2d304cf42d2b686b6c46ab611fd [id=bf47771ababf4cc391d85a9c383690a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.916389 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e17d5171174efd9b628a0cc38b4f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.912292 1155 tablet_bootstrap.cc:654] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.907708 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 989b26c3e2294005a1ebb4316c81ac29 (DEFAULT_TABLE table=default.Table_1591a004d955474d8456f3f4484e8d5f [id=9b59997abe6d438a8722061717ca2276]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.920308 749 tablet_service.cc:1468] Processing CreateTablet for tablet 352a8dbd7f164ca9b622fed30bc6beeb (DEFAULT_TABLE table=default.Table_32c0db573e334edd9afb9efcdff8d7ad [id=93add8c7920d4cec94689ce1e62148ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.920526 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989b26c3e2294005a1ebb4316c81ac29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.920567 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352a8dbd7f164ca9b622fed30bc6beeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.921025 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 86b826e303c14f5ca0af6a336673860b (DEFAULT_TABLE table=default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.921274 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b826e303c14f5ca0af6a336673860b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.922439 884 tablet_service.cc:1468] Processing CreateTablet for tablet df8c1e4fdbc84a08a298d1d5d7c125b3 (DEFAULT_TABLE table=default.Table_6e9154fa65874eb7adab5dcc6ca51d32 [id=5893ea4ae9b84777bfffb839d0b27d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.912335 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 5c534f4fb7354fceb5f50df83292371a (DEFAULT_TABLE table=default.Table_ef090d4a5e444517a98a92efc4d7f581 [id=2518e68e6cf9435f8d8510cabd561ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.922765 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c534f4fb7354fceb5f50df83292371a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.924299 879 tablet_service.cc:1468] Processing CreateTablet for tablet 90eef69522e54b12b849d94e56355921 (DEFAULT_TABLE table=default.Table_b92827e379f148d4bd76b3e578adc695 [id=6faf9bc8cce84174a48928674dc25702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.924602 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 90eef69522e54b12b849d94e56355921 (DEFAULT_TABLE table=default.Table_b92827e379f148d4bd76b3e578adc695 [id=6faf9bc8cce84174a48928674dc25702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.924862 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90eef69522e54b12b849d94e56355921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.912609 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2521a476643c41c1a2e5282755a58091 (DEFAULT_TABLE table=default.Table_c5ad96e3c69d4a29ab6c662b82774b53 [id=35205188f6ee434a822596e1f23a2253]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.926185 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2521a476643c41c1a2e5282755a58091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.927727 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae366e6ccf34a9faf2caa2bded63546 (DEFAULT_TABLE table=default.Table_55565d53ffdd403595e1e5cb1fd7effb [id=a1aced4f425d4711bdea9093f19c791c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.927981 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae366e6ccf34a9faf2caa2bded63546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.928294 881 tablet_service.cc:1468] Processing CreateTablet for tablet 352a8dbd7f164ca9b622fed30bc6beeb (DEFAULT_TABLE table=default.Table_32c0db573e334edd9afb9efcdff8d7ad [id=93add8c7920d4cec94689ce1e62148ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.928660 888 tablet_service.cc:1468] Processing CreateTablet for tablet 7fdabc198d4b48ebb82cd8a889f7dedf (DEFAULT_TABLE table=default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73 [id=4d47d71764394be1bb54e71d9ec9dedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.928944 877 tablet_service.cc:1468] Processing CreateTablet for tablet 86b826e303c14f5ca0af6a336673860b (DEFAULT_TABLE table=default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.929630 883 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae366e6ccf34a9faf2caa2bded63546 (DEFAULT_TABLE table=default.Table_55565d53ffdd403595e1e5cb1fd7effb [id=a1aced4f425d4711bdea9093f19c791c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.930716 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eb8ae50799eb4f4693b4df01e18e82d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:50.931170 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb8ae50799eb4f4693b4df01e18e82d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:50.912887 1019 tablet_service.cc:1468] Processing CreateTablet for tablet f5e17d5171174efd9b628a0cc38b4f49 (DEFAULT_TABLE table=default.Table_629fa2d304cf42d2b686b6c46ab611fd [id=bf47771ababf4cc391d85a9c383690a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.931761 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e17d5171174efd9b628a0cc38b4f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.931753 890 tablet_service.cc:1468] Processing CreateTablet for tablet ac560295fddd44368e6f9bc543f7cbfa (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:50.932294 440 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 543242e4a5064268ae1e5519b569f682 (table default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 543242e4a5064268ae1e5519b569f682 consensus state has no leader
I20250629 01:51:50.932339 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_21c4d86e1513448886bb2c76904e6ef2" } modify_external_catalogs: true
I20250629 01:51:50.932466 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_21c4d86e1513448886bb2c76904e6ef2" } modify_external_catalogs: true
W20250629 01:51:50.932591 1851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:51:50.932986 1851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:51:50.933082 1851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250629 01:51:50.933149 895 tablet_service.cc:1468] Processing CreateTablet for tablet 5160fe759e074d93872a8f0ca7847299 (DEFAULT_TABLE table=default.Table_a092a591616041599f1b07e758ef38d1 [id=15d7913d1c9a436db15cb2b8f7749432]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.935184 743 tablet_service.cc:1468] Processing CreateTablet for tablet 7fdabc198d4b48ebb82cd8a889f7dedf (DEFAULT_TABLE table=default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73 [id=4d47d71764394be1bb54e71d9ec9dedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.935375 745 tablet_service.cc:1468] Processing CreateTablet for tablet 86b826e303c14f5ca0af6a336673860b (DEFAULT_TABLE table=default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.935650 742 tablet_service.cc:1515] Processing DeleteTablet for tablet e1fe3880e51345269e852014d0d60bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:50.935773 739 tablet_service.cc:1468] Processing CreateTablet for tablet 5160fe759e074d93872a8f0ca7847299 (DEFAULT_TABLE table=default.Table_a092a591616041599f1b07e758ef38d1 [id=15d7913d1c9a436db15cb2b8f7749432]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.935887 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b826e303c14f5ca0af6a336673860b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.936029 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5160fe759e074d93872a8f0ca7847299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.935986 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae366e6ccf34a9faf2caa2bded63546 (DEFAULT_TABLE table=default.Table_55565d53ffdd403595e1e5cb1fd7effb [id=a1aced4f425d4711bdea9093f19c791c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.936236 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae366e6ccf34a9faf2caa2bded63546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.940624 740 tablet_service.cc:1468] Processing CreateTablet for tablet ac560295fddd44368e6f9bc543f7cbfa (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.940876 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac560295fddd44368e6f9bc543f7cbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.941313 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fdabc198d4b48ebb82cd8a889f7dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.942885 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7d330f73937f409590ae615703808e30" } modify_external_catalogs: true
I20250629 01:51:50.943012 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7d330f73937f409590ae615703808e30" } modify_external_catalogs: true
I20250629 01:51:50.916236 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 352a8dbd7f164ca9b622fed30bc6beeb (DEFAULT_TABLE table=default.Table_32c0db573e334edd9afb9efcdff8d7ad [id=93add8c7920d4cec94689ce1e62148ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.944427 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352a8dbd7f164ca9b622fed30bc6beeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.916788 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:50.948197 1049 raft_consensus.cc:2466] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:50.918530 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 7fdabc198d4b48ebb82cd8a889f7dedf (DEFAULT_TABLE table=default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73 [id=4d47d71764394be1bb54e71d9ec9dedc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.919901 1021 tablet_service.cc:1468] Processing CreateTablet for tablet df8c1e4fdbc84a08a298d1d5d7c125b3 (DEFAULT_TABLE table=default.Table_6e9154fa65874eb7adab5dcc6ca51d32 [id=5893ea4ae9b84777bfffb839d0b27d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.948552 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fdabc198d4b48ebb82cd8a889f7dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.948541 710 leader_election.cc:304] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:50.948683 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df8c1e4fdbc84a08a298d1d5d7c125b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.952610 1850 raft_consensus.cc:2802] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:50.952677 1850 raft_consensus.cc:491] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:50.952725 1850 raft_consensus.cc:3058] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:50.952798 1159 tablet_bootstrap.cc:492] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:50.952913 1159 ts_tablet_manager.cc:1397] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.148s user 0.001s sys 0.003s
I20250629 01:51:50.954870 1159 raft_consensus.cc:357] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.954990 1159 raft_consensus.cc:383] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.955026 1159 raft_consensus.cc:738] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.955175 1159 consensus_queue.cc:260] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.955343 1850 raft_consensus.cc:513] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:50.955577 1159 ts_tablet_manager.cc:1428] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250629 01:51:50.955785 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:50.956276 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 989b26c3e2294005a1ebb4316c81ac29. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:50.956238 712 leader_election.cc:343] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.956624 1850 leader_election.cc:290] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:50.958179 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 5160fe759e074d93872a8f0ca7847299 (DEFAULT_TABLE table=default.Table_a092a591616041599f1b07e758ef38d1 [id=15d7913d1c9a436db15cb2b8f7749432]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.959228 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:50.959576 712 leader_election.cc:343] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:50.960036 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:50.960225 1048 raft_consensus.cc:3058] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:50.961387 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ac560295fddd44368e6f9bc543f7cbfa (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.961748 746 tablet_service.cc:1468] Processing CreateTablet for tablet df8c1e4fdbc84a08a298d1d5d7c125b3 (DEFAULT_TABLE table=default.Table_6e9154fa65874eb7adab5dcc6ca51d32 [id=5893ea4ae9b84777bfffb839d0b27d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:50.962440 1048 raft_consensus.cc:2466] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:50.962716 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 674) took 1903 ms. Trace:
I20250629 01:51:50.962802 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df8c1e4fdbc84a08a298d1d5d7c125b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.962822 1157 rpcz_store.cc:276] 0629 01:51:49.059037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.072143 (+ 13106us) service_pool.cc:225] Handling call
0629 01:51:50.891023 (+1818880us) ts_tablet_manager.cc:1145] Deleting tablet 77eae224763d464496066b4a98efe5ff
0629 01:51:50.891032 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.962702 (+ 71670us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818748}
I20250629 01:51:50.963212 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 671) took 1909 ms. Trace:
I20250629 01:51:50.963758 1157 rpcz_store.cc:276] 0629 01:51:49.053353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.082549 (+ 29196us) service_pool.cc:225] Handling call
0629 01:51:50.963131 (+1880582us) ts_tablet_manager.cc:1145] Deleting tablet bd5150eff1ad43588a9e70dab47744e3
0629 01:51:50.963138 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.963199 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880446}
I20250629 01:51:50.964282 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 676) took 1773 ms. Trace:
W20250629 01:51:50.965924 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 77eae224763d464496066b4a98efe5ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77eae224763d464496066b4a98efe5ff already in progress: creating tablet
I20250629 01:51:50.962805 710 leader_election.cc:304] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:50.968115 1850 raft_consensus.cc:2802] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:50.968226 1850 raft_consensus.cc:695] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:50.968267 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df8c1e4fdbc84a08a298d1d5d7c125b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.968421 1850 consensus_queue.cc:237] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:50.975762 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 58006.8us
I20250629 01:51:50.964067 1156 ts_tablet_manager.cc:1918] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:50.967288 1155 tablet_bootstrap.cc:492] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:50.977082 1155 ts_tablet_manager.cc:1397] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.000s
I20250629 01:51:50.977778 1155 raft_consensus.cc:357] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.977967 1155 raft_consensus.cc:383] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.978055 1155 raft_consensus.cc:738] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.978261 1155 consensus_queue.cc:260] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.979643 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5160fe759e074d93872a8f0ca7847299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.980080 1155 ts_tablet_manager.cc:1428] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:50.980419 1155 tablet_bootstrap.cc:492] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:50.982511 1155 tablet_bootstrap.cc:654] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:50.984707 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac560295fddd44368e6f9bc543f7cbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.987005 1156 log.cc:1199] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/dd709b557b334434a6c2c4a60a996342
I20250629 01:51:50.987514 1156 ts_tablet_manager.cc:1939] T dd709b557b334434a6c2c4a60a996342 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:50.989038 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet dd709b557b334434a6c2c4a60a996342 (table default.Table_b5f98a84fe7341ac8e6b11b04066323c [id=dd8b9804d0144a879c539614acde7240]) successfully deleted
I20250629 01:51:50.964663 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac560295fddd44368e6f9bc543f7cbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.990023 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90eef69522e54b12b849d94e56355921. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.965056 1159 tablet_bootstrap.cc:492] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:50.986140 884 tablet_service.cc:1515] Processing DeleteTablet for tablet d91fa0f049904a1fb5cbd38d1fd1693a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:50.990180 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5160fe759e074d93872a8f0ca7847299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.990335 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2521a476643c41c1a2e5282755a58091. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.990715 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5e17d5171174efd9b628a0cc38b4f49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.990867 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352a8dbd7f164ca9b622fed30bc6beeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.991011 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c534f4fb7354fceb5f50df83292371a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.991158 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae366e6ccf34a9faf2caa2bded63546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:50.991039 1156 tablet_replica.cc:331] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:50.995431 1156 raft_consensus.cc:2241] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:50.995841 1156 raft_consensus.cc:2270] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:50.996865 1156 ts_tablet_manager.cc:1905] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:50.998729 1154 raft_consensus.cc:357] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.998863 1154 raft_consensus.cc:383] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:50.998905 1154 raft_consensus.cc:738] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:50.999055 1154 consensus_queue.cc:260] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:50.999984 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fdabc198d4b48ebb82cd8a889f7dedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.002866 1155 tablet_bootstrap.cc:492] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.003443 1155 ts_tablet_manager.cc:1397] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.002s
I20250629 01:51:51.003443 1154 ts_tablet_manager.cc:1428] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.102s user 0.002s sys 0.000s
I20250629 01:51:51.004076 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86b826e303c14f5ca0af6a336673860b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.004215 1154 tablet_bootstrap.cc:492] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.004782 1155 raft_consensus.cc:357] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.005448 1154 tablet_bootstrap.cc:654] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.006646 1155 raft_consensus.cc:383] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.006749 1155 raft_consensus.cc:738] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.009128 1159 tablet_bootstrap.cc:654] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.009720 1158 ts_tablet_manager.cc:1918] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250629 01:51:51.013901 1851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250629 01:51:51.014022 1851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250629 01:51:51.014122 1851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250629 01:51:51.014199 1851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250629 01:51:51.016151 1158 log.cc:1199] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e508393b9cb044c4b90194c8aab3184a
I20250629 01:51:51.016496 1158 ts_tablet_manager.cc:1939] T e508393b9cb044c4b90194c8aab3184a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.018127 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]) successfully deleted
I20250629 01:51:51.026904 1915 raft_consensus.cc:491] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.027041 1915 raft_consensus.cc:513] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.027578 1915 leader_election.cc:290] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.027873 1851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=43231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 29 Jun 2025 01:43:20 UTC on 24a791456cd2
build id 6820
ASAN enabled
W20250629 01:51:51.028868 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2aca26af31ed4bc994f4d62603edc881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aca26af31ed4bc994f4d62603edc881 already in progress: creating tablet
W20250629 01:51:51.028869 850 leader_election.cc:343] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:51.029160 1158 tablet_replica.cc:331] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.029376 1158 raft_consensus.cc:2241] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.029541 1158 raft_consensus.cc:2270] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:51.029906 850 leader_election.cc:343] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:51.029968 1158 ts_tablet_manager.cc:1905] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.029999 850 leader_election.cc:304] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.030157 1851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250629 01:51:51.030169 1915 raft_consensus.cc:2747] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.032198 1851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250629 01:51:50.990206 1157 rpcz_store.cc:276] 0629 01:51:49.190486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.190533 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:50.964205 (+1773672us) ts_tablet_manager.cc:1145] Deleting tablet ca13eb6bdcd044ec948ea699857a30f8
0629 01:51:50.964211 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:50.964268 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773535}
I20250629 01:51:51.035286 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 685) took 1817 ms. Trace:
I20250629 01:51:51.035362 1157 rpcz_store.cc:276] 0629 01:51:49.217898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.218183 (+ 285us) service_pool.cc:225] Handling call
0629 01:51:51.035185 (+1817002us) ts_tablet_manager.cc:1145] Deleting tablet 3bdc6a26b3594d6fb33a27cac5b29f4f
0629 01:51:51.035193 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.035273 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1816851}
I20250629 01:51:51.035615 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 684) took 1817 ms. Trace:
I20250629 01:51:51.035681 1157 rpcz_store.cc:276] 0629 01:51:49.217746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.218411 (+ 665us) service_pool.cc:225] Handling call
0629 01:51:51.035537 (+1817126us) ts_tablet_manager.cc:1145] Deleting tablet f94d8fc1b6254ba69acadcaff3613d97
0629 01:51:51.035543 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.035602 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1816920}
I20250629 01:51:51.035872 1157 tablet_replica.cc:331] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.036010 1157 raft_consensus.cc:2241] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.036139 1157 raft_consensus.cc:2270] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.036561 1157 ts_tablet_manager.cc:1905] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.037654 1155 consensus_queue.cc:260] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:51.040400 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ca13eb6bdcd044ec948ea699857a30f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca13eb6bdcd044ec948ea699857a30f8 already in progress: creating tablet
W20250629 01:51:51.040608 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3bdc6a26b3594d6fb33a27cac5b29f4f already in progress: creating tablet
W20250629 01:51:51.040779 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f94d8fc1b6254ba69acadcaff3613d97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f94d8fc1b6254ba69acadcaff3613d97 already in progress: creating tablet
I20250629 01:51:51.042995 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]). Expected version 3 got 0
I20250629 01:51:51.043223 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:51.046104 1155 ts_tablet_manager.cc:1428] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
W20250629 01:51:51.048801 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4609264b54241a0af744585f8bdbc89 consensus state has no leader
I20250629 01:51:51.050432 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_540c577da2d044a19efbc470511fcf47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.051028 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_540c577da2d044a19efbc470511fcf47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.051287 1155 tablet_bootstrap.cc:492] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.052599 1155 tablet_bootstrap.cc:654] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.053366 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 40266441c3684726a58a5029779395fc
I20250629 01:51:51.054018 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cbd53c3618ee416a842d119b503fa368" } new_table_name: "default.Table_beded2997aef4fe0a43329e554eebfea" modify_external_catalogs: true
I20250629 01:51:51.056298 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 40266441c3684726a58a5029779395fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:51.059077 1935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:51.059634 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 543242e4a5064268ae1e5519b569f682
I20250629 01:51:51.060039 449 catalog_manager.cc:6652] ReplaceTablet: tablet 543242e4a5064268ae1e5519b569f682 deleted and replaced by tablet df0fe7f5469d44fc8f6ac5c72377b48a
W20250629 01:51:51.059417 1936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:51.060454 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_709271d3d173470ab9aace403ba9d5e3" } modify_external_catalogs: true
I20250629 01:51:51.060627 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_709271d3d173470ab9aace403ba9d5e3" } modify_external_catalogs: true
I20250629 01:51:51.061431 1158 ts_tablet_manager.cc:1918] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.061527 1158 log.cc:1199] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ea00b4559e03410885180286b5e9fd10
I20250629 01:51:51.061789 1158 ts_tablet_manager.cc:1939] T ea00b4559e03410885180286b5e9fd10 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.056131 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 40266441c3684726a58a5029779395fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.061803 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 543242e4a5064268ae1e5519b569f682 with delete_type TABLET_DATA_DELETED (replaced by tablet df0fe7f5469d44fc8f6ac5c72377b48a) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.061859 1156 ts_tablet_manager.cc:1918] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.062124 1156 log.cc:1199] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/be8e0946a20340b38835c4d8daeaa7b8
I20250629 01:51:51.062206 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 543242e4a5064268ae1e5519b569f682 with delete_type TABLET_DATA_DELETED (replaced by tablet df0fe7f5469d44fc8f6ac5c72377b48a) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.062600 1156 ts_tablet_manager.cc:1939] T be8e0946a20340b38835c4d8daeaa7b8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.063464 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet be8e0946a20340b38835c4d8daeaa7b8 (table default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]) successfully deleted
I20250629 01:51:51.063725 1156 tablet_replica.cc:331] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.063786 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 40266441c3684726a58a5029779395fc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.063891 1156 raft_consensus.cc:2241] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.064007 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 543242e4a5064268ae1e5519b569f682 with delete_type TABLET_DATA_DELETED (replaced by tablet df0fe7f5469d44fc8f6ac5c72377b48a) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.064040 1156 raft_consensus.cc:2270] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.064349 1158 tablet_replica.cc:331] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.064491 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ea00b4559e03410885180286b5e9fd10 (table default.Table_5d35d0525f924ed692497dbaf2287e8e [id=e7ad9b02d0f14bdfa3914844146a1e0b]) successfully deleted
I20250629 01:51:51.064545 1156 ts_tablet_manager.cc:1905] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.066437 1159 tablet_bootstrap.cc:492] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.066592 1159 ts_tablet_manager.cc:1397] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.006s
I20250629 01:51:51.067234 1159 raft_consensus.cc:357] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.067448 1159 raft_consensus.cc:383] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.067534 1159 raft_consensus.cc:738] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.067764 1159 consensus_queue.cc:260] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.068727 1158 raft_consensus.cc:2241] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.069098 1158 raft_consensus.cc:2270] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.069808 1158 ts_tablet_manager.cc:1905] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.072650 1156 ts_tablet_manager.cc:1918] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.073418 1159 ts_tablet_manager.cc:1428] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:51.073621 1156 log.cc:1199] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0a574f95ac2a4193a563e63cbe120768
I20250629 01:51:51.074066 1851 server_base.cc:1048] running on GCE node
I20250629 01:51:51.074580 1156 ts_tablet_manager.cc:1939] T 0a574f95ac2a4193a563e63cbe120768 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.075352 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0a574f95ac2a4193a563e63cbe120768 (table default.Table_399e960d59fe4143843457bd372dbf63 [id=99712a47e4424c78bb4721eeec6e90c3]) successfully deleted
W20250629 01:51:51.075913 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 01d95356c7a7419bb56ddf67d451eadc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01d95356c7a7419bb56ddf67d451eadc already in progress: creating tablet
I20250629 01:51:51.076197 1156 tablet_replica.cc:331] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:51.076321 1940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250629 01:51:51.076385 1156 raft_consensus.cc:2241] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.077140 1159 tablet_bootstrap.cc:492] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.077193 1851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250629 01:51:51.078467 1159 tablet_bootstrap.cc:654] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.081115 1155 tablet_bootstrap.cc:492] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.081283 1155 ts_tablet_manager.cc:1397] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250629 01:51:51.081918 1154 tablet_bootstrap.cc:492] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.082028 1154 ts_tablet_manager.cc:1397] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.009s
I20250629 01:51:51.081936 1155 raft_consensus.cc:357] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.082147 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5ab0aa4e74d345b2a214b90b5018c82b
I20250629 01:51:51.082203 1155 raft_consensus.cc:383] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.082257 1155 raft_consensus.cc:738] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.082434 1155 consensus_queue.cc:260] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.082680 1154 raft_consensus.cc:357] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.082805 1154 raft_consensus.cc:383] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.082842 1154 raft_consensus.cc:738] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.083017 1154 consensus_queue.cc:260] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.083946 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.083984 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.084594 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.085722 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9d377064eeb847699a17f707f0cae83c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.086159 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d377064eeb847699a17f707f0cae83c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.086210 1155 ts_tablet_manager.cc:1428] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:51.087128 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 845f2422d6494c95a4738f3062c216c7
I20250629 01:51:51.087282 1154 ts_tablet_manager.cc:1428] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:51.087535 448 catalog_manager.cc:6652] ReplaceTablet: tablet 845f2422d6494c95a4738f3062c216c7 deleted and replaced by tablet 8ae0d61d254c4e1ba508e30aa35600ca
I20250629 01:51:51.087718 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 9bbbeb43473a4f6f892056a30f62589f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:51.087787 1155 tablet_bootstrap.cc:492] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.088845 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cdb3a884c9ca4f72803e5bc54a78d34a" } new_table_name: "default.Table_c23a34743c38460887967df74d4a4bfb" modify_external_catalogs: true
I20250629 01:51:51.089989 1156 raft_consensus.cc:2270] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.091158 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 845f2422d6494c95a4738f3062c216c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ae0d61d254c4e1ba508e30aa35600ca) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.091199 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 845f2422d6494c95a4738f3062c216c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ae0d61d254c4e1ba508e30aa35600ca) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.091288 1156 ts_tablet_manager.cc:1905] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.091485 1155 tablet_bootstrap.cc:654] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.092387 1154 tablet_bootstrap.cc:492] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.093605 1154 tablet_bootstrap.cc:654] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.093914 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 845f2422d6494c95a4738f3062c216c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8ae0d61d254c4e1ba508e30aa35600ca) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.094872 1158 ts_tablet_manager.cc:1918] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.098433 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5c1d273c70ac4db7a215f02ecac11154
I20250629 01:51:51.099483 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c1d273c70ac4db7a215f02ecac11154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.100170 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8e2c9df275d2453daa8b73a723d56ed6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.100617 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e2c9df275d2453daa8b73a723d56ed6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.103974 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7eb02d3ff1104f6885ea1736852bbe57
I20250629 01:51:51.104543 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c5c42b0786ae4ba8ae6d0165aae1bc60" } new_table_name: "default.Table_8ffd172d7dc7411083dcbc1e695b647d" modify_external_catalogs: true
I20250629 01:51:51.106046 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 7eb02d3ff1104f6885ea1736852bbe57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.100167 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c1d273c70ac4db7a215f02ecac11154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.106176 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 7eb02d3ff1104f6885ea1736852bbe57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.106767 1158 log.cc:1199] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:51.107035 1158 ts_tablet_manager.cc:1939] T 4a692f00ba244a80b9a6d07029f8d27e P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.107985 1158 tablet_replica.cc:331] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.108165 1158 raft_consensus.cc:2241] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.108444 1158 raft_consensus.cc:2270] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.108628 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c1d273c70ac4db7a215f02ecac11154 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.108847 1155 tablet_bootstrap.cc:492] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.108923 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 7eb02d3ff1104f6885ea1736852bbe57 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.109035 1155 ts_tablet_manager.cc:1397] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250629 01:51:51.109100 1158 ts_tablet_manager.cc:1905] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.109675 1155 raft_consensus.cc:357] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.109860 1155 raft_consensus.cc:383] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.109941 1155 raft_consensus.cc:738] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.110036 1157 ts_tablet_manager.cc:1918] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.110155 1155 consensus_queue.cc:260] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.112872 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4a692f00ba244a80b9a6d07029f8d27e (table default.Table_e605f1f30c8d4fdd8503b71227359798 [id=285208f892d84eea88ede0f96db5e7cf]) successfully deleted
W20250629 01:51:51.113086 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 39d2adf801934179825f96a5f7b3c303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39d2adf801934179825f96a5f7b3c303 already in progress: creating tablet
W20250629 01:51:51.113314 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 01d95356c7a7419bb56ddf67d451eadc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01d95356c7a7419bb56ddf67d451eadc already in progress: creating tablet
I20250629 01:51:51.113826 1157 log.cc:1199] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/881461aa1bc34dd483f65e171c341bc5
I20250629 01:51:51.114173 1155 ts_tablet_manager.cc:1428] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:51.115255 1154 tablet_bootstrap.cc:492] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.115358 1154 ts_tablet_manager.cc:1397] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250629 01:51:51.114462 1159 tablet_bootstrap.cc:492] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.115890 1159 ts_tablet_manager.cc:1397] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.001s
I20250629 01:51:51.115981 1154 raft_consensus.cc:357] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.116127 1154 raft_consensus.cc:383] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.116178 1154 raft_consensus.cc:738] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.116369 1154 consensus_queue.cc:260] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.117240 1155 tablet_bootstrap.cc:492] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.117208 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c] alter complete (version 2)
I20250629 01:51:51.117686 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5b6cb001443c433a8a3b15de2a24fd9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.118023 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3adc76548b5247bd99b926acdb4bea43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.118191 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b6cb001443c433a8a3b15de2a24fd9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:51.118441 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3adc76548b5247bd99b926acdb4bea43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.119419 1155 tablet_bootstrap.cc:654] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.119467 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 07e0727102d54294a7c13a4aeef311e1 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:51.119558 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet df66984acfa44a99bf69096aff6d49ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.120195 1154 ts_tablet_manager.cc:1428] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:51.120425 1154 tablet_bootstrap.cc:492] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.121824 1154 tablet_bootstrap.cc:654] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.122114 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_33af2617f1ad46a3b9a25fe7c34f4df6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.122473 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet e83009c1599a43d18d1c73c0101ab0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:51.122498 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33af2617f1ad46a3b9a25fe7c34f4df6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.122577 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 81b32bac5a2144d1943285f5f446d53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.122623 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.125214 1159 raft_consensus.cc:357] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.125373 1159 raft_consensus.cc:383] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.125419 1159 raft_consensus.cc:738] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.125443 1158 ts_tablet_manager.cc:1918] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.125526 1158 log.cc:1199] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/2f7c1f1af39d47c191d34dc026f7f687
I20250629 01:51:51.125754 1158 ts_tablet_manager.cc:1939] T 2f7c1f1af39d47c191d34dc026f7f687 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.127207 1154 tablet_bootstrap.cc:492] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.127337 1154 ts_tablet_manager.cc:1397] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250629 01:51:51.130781 1154 raft_consensus.cc:357] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.130920 1154 raft_consensus.cc:383] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.130977 1154 raft_consensus.cc:738] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.131162 1154 consensus_queue.cc:260] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:51.131533 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4a3b7d971694e15be1aaa072468dae2 consensus state has no leader
I20250629 01:51:51.133304 1851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250629 01:51:51.134066 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c35ba7564d5a4e53bbe91f2b50ac8ad5
I20250629 01:51:51.134177 1154 ts_tablet_manager.cc:1428] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250629 01:51:51.134688 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_815cc062f73f433da58e8a2fcc368854" } new_table_name: "default.Table_355a96774d4f4e0e9f5835f7a7450e0c" modify_external_catalogs: true
I20250629 01:51:51.134871 877 tablet_service.cc:1515] Processing DeleteTablet for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.135182 1850 raft_consensus.cc:491] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.135294 1850 raft_consensus.cc:513] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.135459 1156 ts_tablet_manager.cc:1918] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.135568 1156 log.cc:1199] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/77eae224763d464496066b4a98efe5ff
I20250629 01:51:51.135696 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9bbbeb43473a4f6f892056a30f62589f
I20250629 01:51:51.135816 1850 leader_election.cc:290] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.136117 448 catalog_manager.cc:6652] ReplaceTablet: tablet 9bbbeb43473a4f6f892056a30f62589f deleted and replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1
I20250629 01:51:51.136387 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1fe3880e51345269e852014d0d60bb6" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.136474 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e50b86b187294b3fb9f068be98e6101d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.136600 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bbbeb43473a4f6f892056a30f62589f with delete_type TABLET_DATA_DELETED (replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:51.136829 712 leader_election.cc:343] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:51.136878 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e50b86b187294b3fb9f068be98e6101d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.135977 1156 ts_tablet_manager.cc:1939] T 77eae224763d464496066b4a98efe5ff P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.137614 1154 tablet_bootstrap.cc:492] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:51.137892 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bb4d7c2096a45df8eb392a2e7d85c3d consensus state has no leader
I20250629 01:51:51.138156 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 2540a84a922949e3adb90d0cded1c951 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:51.138413 1923 raft_consensus.cc:491] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:51.138504 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4cc7e839241e447db039b261421a7fef consensus state has no leader
I20250629 01:51:51.138504 1156 tablet_replica.cc:331] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.138509 1923 raft_consensus.cc:513] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.138900 1156 raft_consensus.cc:2241] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.138990 1923 leader_election.cc:290] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.139081 1156 raft_consensus.cc:2270] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.140441 1156 ts_tablet_manager.cc:1905] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.141516 1851 hybrid_clock.cc:648] HybridClock initialized: now 1751161911141409 us; error 91 us; skew 500 ppm
I20250629 01:51:51.141906 1851 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250629 01:51:51.143359 1155 tablet_bootstrap.cc:492] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.143540 1155 ts_tablet_manager.cc:1397] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20250629 01:51:51.143865 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541728ff8dc3471ba90784b4a769c8e1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:51.144219 1155 raft_consensus.cc:357] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:51.144268 986 leader_election.cc:343] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.144379 1155 raft_consensus.cc:383] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.144444 1155 raft_consensus.cc:738] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.144681 1154 tablet_bootstrap.cc:654] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.144692 1155 consensus_queue.cc:260] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.145722 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c951e1bffffa4ed7b7fcff1b87d9ee23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.146600 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c951e1bffffa4ed7b7fcff1b87d9ee23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.147410 1158 tablet_replica.cc:331] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:51.147632 710 leader_election.cc:343] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e1fe3880e51345269e852014d0d60bb6
I20250629 01:51:51.147747 710 leader_election.cc:304] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:51.147781 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541728ff8dc3471ba90784b4a769c8e1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.148029 1158 raft_consensus.cc:2241] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.148034 1850 raft_consensus.cc:2747] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.148166 1158 raft_consensus.cc:2270] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:51.148366 988 leader_election.cc:343] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.148475 988 leader_election.cc:304] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.148583 1158 ts_tablet_manager.cc:1905] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.148686 1923 raft_consensus.cc:2747] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.149705 1155 ts_tablet_manager.cc:1428] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:51:51.152837 1155 tablet_bootstrap.cc:492] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.152882 1154 tablet_bootstrap.cc:492] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.153028 1154 ts_tablet_manager.cc:1397] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20250629 01:51:51.153743 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.154165 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.154314 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bbbeb43473a4f6f892056a30f62589f with delete_type TABLET_DATA_DELETED (replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.154515 1154 raft_consensus.cc:357] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.154697 1154 raft_consensus.cc:383] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.154733 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bbbeb43473a4f6f892056a30f62589f with delete_type TABLET_DATA_DELETED (replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.154785 1154 raft_consensus.cc:738] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.154950 1154 consensus_queue.cc:260] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.158732 1154 ts_tablet_manager.cc:1428] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250629 01:51:51.159116 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 2f7c1f1af39d47c191d34dc026f7f687 (table default.Table_f641040c5d3f4fabb4642ae6c32b10c7 [id=7607f62687244e718732520ce1342b1a]) successfully deleted
W20250629 01:51:51.159340 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2d37334db2b9465a8fcf5e849d5d3b0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d37334db2b9465a8fcf5e849d5d3b0c already in progress: creating tablet
W20250629 01:51:51.159511 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f52937830a924d6d8338d78da24b84cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52937830a924d6d8338d78da24b84cd already in progress: creating tablet
W20250629 01:51:51.159662 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 40266441c3684726a58a5029779395fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40266441c3684726a58a5029779395fc already in progress: creating tablet
I20250629 01:51:51.159780 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 77eae224763d464496066b4a98efe5ff (table default.Table_6f7d1b03023842daa7f9d43483dc6b95 [id=92be86eee6b64fb0984d8e5ea727fbdf]) successfully deleted
W20250629 01:51:51.159938 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2d37334db2b9465a8fcf5e849d5d3b0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d37334db2b9465a8fcf5e849d5d3b0c already in progress: creating tablet
W20250629 01:51:51.160118 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f52937830a924d6d8338d78da24b84cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52937830a924d6d8338d78da24b84cd already in progress: creating tablet
I20250629 01:51:51.161023 1154 tablet_bootstrap.cc:492] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.161679 1159 consensus_queue.cc:260] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.162266 1159 ts_tablet_manager.cc:1428] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.046s user 0.001s sys 0.000s
I20250629 01:51:51.164247 1159 tablet_bootstrap.cc:492] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.165752 1157 ts_tablet_manager.cc:1939] T 881461aa1bc34dd483f65e171c341bc5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.166617 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 686) took 1926 ms. Trace:
I20250629 01:51:51.167086 1157 rpcz_store.cc:276] 0629 01:51:49.240611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.240654 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:51.035839 (+1795185us) ts_tablet_manager.cc:1145] Deleting tablet 881461aa1bc34dd483f65e171c341bc5
0629 01:51:51.035847 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.039083 (+ 3236us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.109963 (+ 70880us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.166575 (+ 56612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795050}
I20250629 01:51:51.167351 1915 raft_consensus.cc:491] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.167034 1159 tablet_bootstrap.cc:654] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.167451 1915 raft_consensus.cc:513] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.167953 1915 leader_election.cc:290] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.168345 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 703) took 1776 ms. Trace:
I20250629 01:51:51.168409 1157 rpcz_store.cc:276] 0629 01:51:49.391675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.392087 (+ 412us) service_pool.cc:225] Handling call
0629 01:51:51.168254 (+1776167us) ts_tablet_manager.cc:1145] Deleting tablet 0bef2747f0c04f498079906e40bac24c
0629 01:51:51.168262 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.168330 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775272}
I20250629 01:51:51.168685 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 704) took 1757 ms. Trace:
I20250629 01:51:51.168744 1157 rpcz_store.cc:276] 0629 01:51:49.410760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.410878 (+ 118us) service_pool.cc:225] Handling call
0629 01:51:51.168606 (+1757728us) ts_tablet_manager.cc:1145] Deleting tablet 22567416bfde4e2b843cf5c7a214ec6c
0629 01:51:51.168613 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.168670 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1755054}
I20250629 01:51:51.168972 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 705) took 1746 ms. Trace:
I20250629 01:51:51.169031 1157 rpcz_store.cc:276] 0629 01:51:49.422233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.422276 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:51.168895 (+1746619us) ts_tablet_manager.cc:1145] Deleting tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b
0629 01:51:51.168903 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.168960 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1737036}
I20250629 01:51:51.169262 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 706) took 1740 ms. Trace:
I20250629 01:51:51.169337 1157 rpcz_store.cc:276] 0629 01:51:49.428341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.428388 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:51.169189 (+1740801us) ts_tablet_manager.cc:1145] Deleting tablet 4aa8dfd712c043ad9f8ff56b064bb8fd
0629 01:51:51.169195 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.169249 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1724799}
I20250629 01:51:51.169543 1157 tablet_replica.cc:331] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:51.169646 850 leader_election.cc:343] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 8a60215417c34bedafbf12172eab469a
I20250629 01:51:51.169674 1157 raft_consensus.cc:2241] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.169773 1157 raft_consensus.cc:2270] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:51.169771 850 leader_election.cc:343] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8a60215417c34bedafbf12172eab469a
I20250629 01:51:51.169955 850 leader_election.cc:304] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.170178 1157 ts_tablet_manager.cc:1905] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.170351 1915 raft_consensus.cc:2747] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.171440 1156 ts_tablet_manager.cc:1918] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.173614 1923 raft_consensus.cc:491] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.173710 1923 raft_consensus.cc:513] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.174146 1923 leader_election.cc:290] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:51.175694 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.175784 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c2b04ca6a212457b94cd696da4139933"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.176203 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2b04ca6a212457b94cd696da4139933 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:51.176486 988 leader_election.cc:343] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.174612 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:51.174732 1850 raft_consensus.cc:491] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.176748 1850 raft_consensus.cc:513] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:51.177037 986 leader_election.cc:343] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.177111 986 leader_election.cc:304] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.177230 1850 leader_election.cc:290] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.177561 1923 raft_consensus.cc:2747] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.177659 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a792a26725954adea15ad57e31bfb224" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:51.177789 1048 raft_consensus.cc:2466] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:51.178107 710 leader_election.cc:304] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:51.178622 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a792a26725954adea15ad57e31bfb224" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.178918 1850 raft_consensus.cc:2802] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:51.178975 1850 raft_consensus.cc:491] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:51.179016 1850 raft_consensus.cc:3058] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:51.179826 712 leader_election.cc:343] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.181525 1850 raft_consensus.cc:513] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.181968 1850 leader_election.cc:290] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.182019 1154 tablet_bootstrap.cc:654] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.182538 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a792a26725954adea15ad57e31bfb224" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:51.182686 1048 raft_consensus.cc:3058] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.183605 1158 ts_tablet_manager.cc:1918] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.183672 1158 log.cc:1199] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/543242e4a5064268ae1e5519b569f682
I20250629 01:51:51.183876 1158 ts_tablet_manager.cc:1939] T 543242e4a5064268ae1e5519b569f682 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.184792 1157 ts_tablet_manager.cc:1918] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.184871 1157 log.cc:1199] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/96324ddf32a1460b8fc37e2c324649e6
I20250629 01:51:51.184895 1158 tablet_replica.cc:331] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.184962 1048 raft_consensus.cc:2466] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:51.185031 1158 raft_consensus.cc:2241] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.185093 1157 ts_tablet_manager.cc:1939] T 96324ddf32a1460b8fc37e2c324649e6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.185145 1158 raft_consensus.cc:2270] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.185492 710 leader_election.cc:304] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:51.185679 1824 raft_consensus.cc:491] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.185762 1824 raft_consensus.cc:513] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.186226 1824 leader_election.cc:290] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.186368 1824 raft_consensus.cc:2802] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:51.186467 1824 raft_consensus.cc:695] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:51.186658 1824 consensus_queue.cc:237] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.186867 1158 ts_tablet_manager.cc:1905] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.189200 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "49333302a27f4ac3b76a1e492b3c1b60" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.189474 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a792a26725954adea15ad57e31bfb224" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:51.189924 712 leader_election.cc:343] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:51.190075 712 leader_election.cc:343] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.190500 1923 raft_consensus.cc:491] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.190596 1923 raft_consensus.cc:513] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:51.190755 710 leader_election.cc:343] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:51.190887 710 leader_election.cc:304] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:51.190979 1923 leader_election.cc:290] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:51.191562 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:51.191597 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.191973 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 708) took 1399 ms. Trace:
W20250629 01:51:51.192011 988 leader_election.cc:343] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:51.192152 986 leader_election.cc:343] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.192294 986 leader_election.cc:304] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.192719 1923 raft_consensus.cc:2747] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.192613 1157 rpcz_store.cc:276] 0629 01:51:49.792019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.792863 (+ 844us) service_pool.cc:225] Handling call
0629 01:51:51.169501 (+1376638us) ts_tablet_manager.cc:1145] Deleting tablet 96324ddf32a1460b8fc37e2c324649e6
0629 01:51:51.169507 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.176442 (+ 6935us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.184725 (+ 8283us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.191959 (+ 7234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1376494}
I20250629 01:51:51.193006 1850 raft_consensus.cc:2747] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.193315 1157 tablet_replica.cc:331] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.193718 1157 raft_consensus.cc:2241] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.194121 1157 raft_consensus.cc:2270] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.194420 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 543242e4a5064268ae1e5519b569f682 (table default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]) successfully deleted
W20250629 01:51:51.194614 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5ab0aa4e74d345b2a214b90b5018c82b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ab0aa4e74d345b2a214b90b5018c82b already in progress: creating tablet
W20250629 01:51:51.194790 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 845f2422d6494c95a4738f3062c216c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 845f2422d6494c95a4738f3062c216c7 already in progress: creating tablet
W20250629 01:51:51.194976 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5c1d273c70ac4db7a215f02ecac11154 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c1d273c70ac4db7a215f02ecac11154 already in progress: creating tablet
W20250629 01:51:51.195180 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7eb02d3ff1104f6885ea1736852bbe57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eb02d3ff1104f6885ea1736852bbe57 already in progress: creating tablet
I20250629 01:51:51.195199 1157 ts_tablet_manager.cc:1905] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.199332 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 881461aa1bc34dd483f65e171c341bc5 (table default.Table_16db51abc1404d7fa67f1c00a70f9166 [id=a922c0f8e123451fb4060f61472e8d41]) successfully deleted
W20250629 01:51:51.200276 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0bef2747f0c04f498079906e40bac24c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0bef2747f0c04f498079906e40bac24c already in progress: creating tablet
W20250629 01:51:51.200898 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 22567416bfde4e2b843cf5c7a214ec6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22567416bfde4e2b843cf5c7a214ec6c already in progress: creating tablet
W20250629 01:51:51.201550 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b already in progress: creating tablet
W20250629 01:51:51.199433 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95247cc778504f4d85d9013f84c59e69 consensus state has no leader
W20250629 01:51:51.202270 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa8dfd712c043ad9f8ff56b064bb8fd already in progress: creating tablet
I20250629 01:51:51.202484 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 96324ddf32a1460b8fc37e2c324649e6 (table default.Table_8c1fbbaabe004466858cce35b18194b1 [id=700ef08e094b44369154076e65d9ea94]) successfully deleted
I20250629 01:51:51.203235 1158 ts_tablet_manager.cc:1918] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.203200 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6d7afa4be82a4ae3974bd0b4786d22db (DEFAULT_TABLE table=default.Table_14492ecf17b84f3f8b5fb91233a242dd [id=9ef021104e3a43f491a515b13e83de16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.203550 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7afa4be82a4ae3974bd0b4786d22db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.204066 1158 log.cc:1199] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:51.204478 1158 ts_tablet_manager.cc:1939] T df66984acfa44a99bf69096aff6d49ab P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.205348 1158 tablet_replica.cc:331] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.205623 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 6d7afa4be82a4ae3974bd0b4786d22db (DEFAULT_TABLE table=default.Table_14492ecf17b84f3f8b5fb91233a242dd [id=9ef021104e3a43f491a515b13e83de16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.205989 743 tablet_service.cc:1468] Processing CreateTablet for tablet 6d7afa4be82a4ae3974bd0b4786d22db (DEFAULT_TABLE table=default.Table_14492ecf17b84f3f8b5fb91233a242dd [id=9ef021104e3a43f491a515b13e83de16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.206212 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7afa4be82a4ae3974bd0b4786d22db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.206311 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7afa4be82a4ae3974bd0b4786d22db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.206459 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 8992da4b777d400b9c3f242444229353 (DEFAULT_TABLE table=default.Table_a7f78eef06dd413cad0a63d1923e1263 [id=e7b5b364a6b0493c8fe9a4008f1ad26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.206724 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8992da4b777d400b9c3f242444229353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.208282 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a2ce14bc48f24499ad1a02fbd53fbe08 (DEFAULT_TABLE table=default.Table_da43c1c1f1c948849cef128c6cbaccfe [id=1f0ee823b98f41daa5ab4ad547198e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.208532 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ce14bc48f24499ad1a02fbd53fbe08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.210307 877 tablet_service.cc:1468] Processing CreateTablet for tablet a2ce14bc48f24499ad1a02fbd53fbe08 (DEFAULT_TABLE table=default.Table_da43c1c1f1c948849cef128c6cbaccfe [id=1f0ee823b98f41daa5ab4ad547198e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.210556 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ce14bc48f24499ad1a02fbd53fbe08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.211341 1158 raft_consensus.cc:2241] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.211644 1158 raft_consensus.cc:2270] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.212278 1029 tablet_service.cc:1468] Processing CreateTablet for tablet d1131e0d9c06435fb732a8a72e421298 (DEFAULT_TABLE table=default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b [id=08fd495cf9b34113a8f228955788d270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.212550 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1131e0d9c06435fb732a8a72e421298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.212652 746 tablet_service.cc:1468] Processing CreateTablet for tablet a2ce14bc48f24499ad1a02fbd53fbe08 (DEFAULT_TABLE table=default.Table_da43c1c1f1c948849cef128c6cbaccfe [id=1f0ee823b98f41daa5ab4ad547198e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.212915 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ce14bc48f24499ad1a02fbd53fbe08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.216307 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8992da4b777d400b9c3f242444229353 (DEFAULT_TABLE table=default.Table_a7f78eef06dd413cad0a63d1923e1263 [id=e7b5b364a6b0493c8fe9a4008f1ad26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.216576 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8992da4b777d400b9c3f242444229353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.216782 881 tablet_service.cc:1468] Processing CreateTablet for tablet 8992da4b777d400b9c3f242444229353 (DEFAULT_TABLE table=default.Table_a7f78eef06dd413cad0a63d1923e1263 [id=e7b5b364a6b0493c8fe9a4008f1ad26a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.217134 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8992da4b777d400b9c3f242444229353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.220324 740 tablet_service.cc:1468] Processing CreateTablet for tablet d1131e0d9c06435fb732a8a72e421298 (DEFAULT_TABLE table=default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b [id=08fd495cf9b34113a8f228955788d270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.220366 1923 raft_consensus.cc:491] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.220491 1923 raft_consensus.cc:513] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.220582 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1131e0d9c06435fb732a8a72e421298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.220611 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 58859bbae17a425fa9bd71bf7656d478 (DEFAULT_TABLE table=default.Table_2ebed88287e94163bf75e1ee852ecc4e [id=963f39507b894b03a43471abf8dc84af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.220844 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58859bbae17a425fa9bd71bf7656d478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.220932 1923 leader_election.cc:290] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:51.220908 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a873bc03533a405185eca6ad4bb5db3a (DEFAULT_TABLE table=default.Table_93f95da30d034c31b5c7f2c52229fcf2 [id=24a9599286bc48a99301a30ce29f1758]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.221175 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a873bc03533a405185eca6ad4bb5db3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.221544 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.221880 988 leader_election.cc:343] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.224258 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:51.224630 986 leader_election.cc:343] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.224713 986 leader_election.cc:304] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.225023 1923 raft_consensus.cc:2747] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.225884 739 tablet_service.cc:1468] Processing CreateTablet for tablet 58859bbae17a425fa9bd71bf7656d478 (DEFAULT_TABLE table=default.Table_2ebed88287e94163bf75e1ee852ecc4e [id=963f39507b894b03a43471abf8dc84af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.226130 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58859bbae17a425fa9bd71bf7656d478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.227807 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0625c1d047504984b8d271004d40f15c (DEFAULT_TABLE table=default.Table_d57756db239b4178a430514058de8274 [id=327c0289637e41a7a4e8d334a205b828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.227881 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet df66984acfa44a99bf69096aff6d49ab (table default.Table_abc4bc33793d4bee824f97f24a172593 [id=8a6a13c4f09547aa8ca713ab9892f557]) successfully deleted
I20250629 01:51:51.228083 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0625c1d047504984b8d271004d40f15c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.228307 738 tablet_service.cc:1468] Processing CreateTablet for tablet a873bc03533a405185eca6ad4bb5db3a (DEFAULT_TABLE table=default.Table_93f95da30d034c31b5c7f2c52229fcf2 [id=24a9599286bc48a99301a30ce29f1758]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.228387 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 35990d206ba64159ab05ab7b7b19d064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.228555 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a873bc03533a405185eca6ad4bb5db3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.229702 881 tablet_service.cc:1468] Processing CreateTablet for tablet d1131e0d9c06435fb732a8a72e421298 (DEFAULT_TABLE table=default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b [id=08fd495cf9b34113a8f228955788d270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.229936 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1131e0d9c06435fb732a8a72e421298. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.231568 1158 ts_tablet_manager.cc:1905] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.233098 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 730178bc660f4af596007721df96198f (DEFAULT_TABLE table=default.Table_bb9d3aa0a3084c80a33203f45b4978eb [id=e3dc00943509471cb5142019e78cf587]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.233160 454 catalog_manager.cc:5582] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:51.233381 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 730178bc660f4af596007721df96198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.233579 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:51.234583 1850 raft_consensus.cc:491] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.234681 1850 raft_consensus.cc:513] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.235006 755 tablet_service.cc:1468] Processing CreateTablet for tablet 730178bc660f4af596007721df96198f (DEFAULT_TABLE table=default.Table_bb9d3aa0a3084c80a33203f45b4978eb [id=e3dc00943509471cb5142019e78cf587]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.235092 1850 leader_election.cc:290] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.235344 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 730178bc660f4af596007721df96198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.235764 1031 tablet_service.cc:1468] Processing CreateTablet for tablet ef95de81b1de41d29a5db6dcdd5dd270 (DEFAULT_TABLE table=default.Table_e67aa47b96b74730985e975276f3745b [id=9aff10a52a83498db82b00099a2f8403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.235946 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f0ae4c51924370896c624affde6761" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.236085 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f2336702c5e941ce994df6edb1970192 (DEFAULT_TABLE table=default.Table_1f13038d42254b24858c6b4d8af9d70a [id=bc7f7f3d60244d8d804384461aa89871]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:51.236517 712 leader_election.cc:343] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:51.237190 710 leader_election.cc:343] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 69f0ae4c51924370896c624affde6761
I20250629 01:51:51.237314 710 leader_election.cc:304] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:51.237847 1861 raft_consensus.cc:2747] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.239565 890 tablet_service.cc:1468] Processing CreateTablet for tablet a873bc03533a405185eca6ad4bb5db3a (DEFAULT_TABLE table=default.Table_93f95da30d034c31b5c7f2c52229fcf2 [id=24a9599286bc48a99301a30ce29f1758]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.239810 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a873bc03533a405185eca6ad4bb5db3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.240125 878 tablet_service.cc:1515] Processing DeleteTablet for tablet dd709b557b334434a6c2c4a60a996342 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.240360 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0625c1d047504984b8d271004d40f15c (DEFAULT_TABLE table=default.Table_d57756db239b4178a430514058de8274 [id=327c0289637e41a7a4e8d334a205b828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.240670 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0625c1d047504984b8d271004d40f15c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.240998 878 tablet_service.cc:1468] Processing CreateTablet for tablet ef95de81b1de41d29a5db6dcdd5dd270 (DEFAULT_TABLE table=default.Table_e67aa47b96b74730985e975276f3745b [id=9aff10a52a83498db82b00099a2f8403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.241338 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef95de81b1de41d29a5db6dcdd5dd270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.244372 745 tablet_service.cc:1468] Processing CreateTablet for tablet ef95de81b1de41d29a5db6dcdd5dd270 (DEFAULT_TABLE table=default.Table_e67aa47b96b74730985e975276f3745b [id=9aff10a52a83498db82b00099a2f8403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.244693 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef95de81b1de41d29a5db6dcdd5dd270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.244062 1157 ts_tablet_manager.cc:1918] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.247660 1157 log.cc:1199] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/75fd4ffcf896419b81a1c0eb7c5aeeec
I20250629 01:51:51.247905 1157 ts_tablet_manager.cc:1939] T 75fd4ffcf896419b81a1c0eb7c5aeeec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.247965 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef95de81b1de41d29a5db6dcdd5dd270. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.248015 881 tablet_service.cc:1468] Processing CreateTablet for tablet 58859bbae17a425fa9bd71bf7656d478 (DEFAULT_TABLE table=default.Table_2ebed88287e94163bf75e1ee852ecc4e [id=963f39507b894b03a43471abf8dc84af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.248311 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2336702c5e941ce994df6edb1970192. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.248312 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58859bbae17a425fa9bd71bf7656d478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.248309 754 tablet_service.cc:1468] Processing CreateTablet for tablet f2336702c5e941ce994df6edb1970192 (DEFAULT_TABLE table=default.Table_1f13038d42254b24858c6b4d8af9d70a [id=bc7f7f3d60244d8d804384461aa89871]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.248497 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 707) took 1456 ms. Trace:
I20250629 01:51:51.248551 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2336702c5e941ce994df6edb1970192. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.248570 1157 rpcz_store.cc:276] 0629 01:51:49.791917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.891027 (+ 99110us) service_pool.cc:225] Handling call
0629 01:51:51.193271 (+1302244us) ts_tablet_manager.cc:1145] Deleting tablet 75fd4ffcf896419b81a1c0eb7c5aeeec
0629 01:51:51.193279 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.241639 (+ 48360us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.244005 (+ 2366us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.248480 (+ 4475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1302030}
I20250629 01:51:51.252377 888 tablet_service.cc:1468] Processing CreateTablet for tablet 0625c1d047504984b8d271004d40f15c (DEFAULT_TABLE table=default.Table_d57756db239b4178a430514058de8274 [id=327c0289637e41a7a4e8d334a205b828]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.252624 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0625c1d047504984b8d271004d40f15c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.252979 1156 log.cc:1199] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/86832a3c26224c7ea0b9b4b6866d3880
I20250629 01:51:51.253258 1156 ts_tablet_manager.cc:1939] T 86832a3c26224c7ea0b9b4b6866d3880 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.254809 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 719) took 1355 ms. Trace:
I20250629 01:51:51.254868 1157 rpcz_store.cc:276] 0629 01:51:49.899667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.899719 (+ 52us) service_pool.cc:225] Handling call
0629 01:51:51.254720 (+1355001us) ts_tablet_manager.cc:1145] Deleting tablet 97d0fbb691514d51a1ca71f32a666c2d
0629 01:51:51.254728 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.254796 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1354818}
I20250629 01:51:51.255582 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 720) took 1355 ms. Trace:
I20250629 01:51:51.255638 1157 rpcz_store.cc:276] 0629 01:51:49.899743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.899918 (+ 175us) service_pool.cc:225] Handling call
0629 01:51:51.255490 (+1355572us) ts_tablet_manager.cc:1145] Deleting tablet 02bd46ceac1e4766ada4c4d2e95fda00
0629 01:51:51.255497 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.255569 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355457}
I20250629 01:51:51.256536 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 721) took 1356 ms. Trace:
I20250629 01:51:51.256594 1157 rpcz_store.cc:276] 0629 01:51:49.899814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.900044 (+ 230us) service_pool.cc:225] Handling call
0629 01:51:51.256212 (+1356168us) ts_tablet_manager.cc:1145] Deleting tablet dc841eb171d04b118b1e705ceed4c861
0629 01:51:51.256442 (+ 230us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.256516 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1355989}
I20250629 01:51:51.256817 896 tablet_service.cc:1468] Processing CreateTablet for tablet 730178bc660f4af596007721df96198f (DEFAULT_TABLE table=default.Table_bb9d3aa0a3084c80a33203f45b4978eb [id=e3dc00943509471cb5142019e78cf587]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.260571 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 730178bc660f4af596007721df96198f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.260558 1158 ts_tablet_manager.cc:1918] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.260654 1158 log.cc:1199] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:51.260877 1158 ts_tablet_manager.cc:1939] T e83009c1599a43d18d1c73c0101ab0be P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.261724 1158 tablet_replica.cc:331] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.261854 1158 raft_consensus.cc:2241] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.261946 1158 raft_consensus.cc:2270] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.262910 1154 tablet_bootstrap.cc:492] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.263021 1154 ts_tablet_manager.cc:1397] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.102s user 0.014s sys 0.015s
I20250629 01:51:51.263636 1154 raft_consensus.cc:357] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.263769 1154 raft_consensus.cc:383] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.263818 1154 raft_consensus.cc:738] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.263988 1154 consensus_queue.cc:260] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.264568 1154 ts_tablet_manager.cc:1428] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:51.265054 877 tablet_service.cc:1468] Processing CreateTablet for tablet f2336702c5e941ce994df6edb1970192 (DEFAULT_TABLE table=default.Table_1f13038d42254b24858c6b4d8af9d70a [id=bc7f7f3d60244d8d804384461aa89871]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.265771 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 75fd4ffcf896419b81a1c0eb7c5aeeec (table default.Table_5ce4c958f3d4456fb56450ca4cc26221 [id=646791dd9bb447fdb2e96a9046dd68d2]) successfully deleted
W20250629 01:51:51.266022 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 97d0fbb691514d51a1ca71f32a666c2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d0fbb691514d51a1ca71f32a666c2d already in progress: creating tablet
W20250629 01:51:51.266191 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 02bd46ceac1e4766ada4c4d2e95fda00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02bd46ceac1e4766ada4c4d2e95fda00 already in progress: creating tablet
W20250629 01:51:51.266395 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet dc841eb171d04b118b1e705ceed4c861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc841eb171d04b118b1e705ceed4c861 already in progress: creating tablet
I20250629 01:51:51.268409 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2c7fd136d3344a2f86e67efda6da2803" } new_table_name: "default.Table_9a5f762a18104801991847cafb8e0e97" modify_external_catalogs: true
I20250629 01:51:51.269887 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2336702c5e941ce994df6edb1970192. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.270397 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73" } modify_external_catalogs: true
I20250629 01:51:51.270553 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c1a06d7a04ed41b597f9b9eb9bea4e73" } modify_external_catalogs: true
I20250629 01:51:51.270823 1154 tablet_bootstrap.cc:492] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.270905 1156 tablet_replica.cc:331] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.271039 1156 raft_consensus.cc:2241] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.271150 1156 raft_consensus.cc:2270] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.272082 1154 tablet_bootstrap.cc:654] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.279923 1155 tablet_bootstrap.cc:700] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Time spent opening tablet: real 0.126s user 0.010s sys 0.000s
I20250629 01:51:51.280180 1155 tablet_bootstrap.cc:654] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.279789 1158 ts_tablet_manager.cc:1905] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.280208 1156 ts_tablet_manager.cc:1905] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.284644 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_82be402325034e178ca2423c9729a2dd" } modify_external_catalogs: true
I20250629 01:51:51.284812 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_82be402325034e178ca2423c9729a2dd" } modify_external_catalogs: true
I20250629 01:51:51.285698 1154 tablet_bootstrap.cc:492] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.285830 1154 ts_tablet_manager.cc:1397] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20250629 01:51:51.287161 1154 raft_consensus.cc:357] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.287328 1154 raft_consensus.cc:383] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.287402 1154 raft_consensus.cc:738] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.287669 1154 consensus_queue.cc:260] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.288975 1156 ts_tablet_manager.cc:1918] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.290060 1156 log.cc:1199] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:51.290392 1156 ts_tablet_manager.cc:1939] T 49333302a27f4ac3b76a1e492b3c1b60 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.290977 1155 tablet_bootstrap.cc:492] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.291463 1155 ts_tablet_manager.cc:1397] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.139s user 0.013s sys 0.006s
I20250629 01:51:51.291736 1154 ts_tablet_manager.cc:1428] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250629 01:51:51.292253 1155 raft_consensus.cc:357] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.292444 1155 raft_consensus.cc:383] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.292526 1155 raft_consensus.cc:738] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.292716 1155 consensus_queue.cc:260] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.295169 1154 tablet_bootstrap.cc:492] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.296144 1155 ts_tablet_manager.cc:1428] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:51.297022 1154 tablet_bootstrap.cc:654] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.298873 1155 tablet_bootstrap.cc:492] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.299100 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3bd1be2b90da48c7aed1e37d5508c9c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.299566 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bd1be2b90da48c7aed1e37d5508c9c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.300391 1155 tablet_bootstrap.cc:654] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.302542 1156 tablet_replica.cc:331] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.302726 1156 raft_consensus.cc:2241] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.302855 1156 raft_consensus.cc:2270] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.303483 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fb9fc3beabf04d8d94b0b64c4e12139d" } modify_external_catalogs: true
I20250629 01:51:51.303625 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fb9fc3beabf04d8d94b0b64c4e12139d" } modify_external_catalogs: true
I20250629 01:51:51.303683 1156 ts_tablet_manager.cc:1905] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.306552 1155 tablet_bootstrap.cc:492] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.306497 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5468f28974964427aa77e0d4809ca02c" } modify_external_catalogs: true
I20250629 01:51:51.306650 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5468f28974964427aa77e0d4809ca02c" } modify_external_catalogs: true
I20250629 01:51:51.306702 1155 ts_tablet_manager.cc:1397] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250629 01:51:51.307394 1155 raft_consensus.cc:357] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.307503 1157 tablet_replica.cc:331] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.307556 1155 raft_consensus.cc:383] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.307672 1155 raft_consensus.cc:738] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.307664 1157 raft_consensus.cc:2241] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.307785 1157 raft_consensus.cc:2270] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.307907 1155 consensus_queue.cc:260] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.310043 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 432680cae61946af9859760fe23110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.310520 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.310956 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 86832a3c26224c7ea0b9b4b6866d3880 (table default.Table_f0a26d9ea3e8407da5ca0813db26beb3 [id=b52a4204ccf34819b3782aca3ea1db0f]) successfully deleted
I20250629 01:51:51.311560 1157 ts_tablet_manager.cc:1905] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.312065 1155 ts_tablet_manager.cc:1428] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:51.312299 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 49333302a27f4ac3b76a1e492b3c1b60 (table default.Table_3ca4f32bc0954a7c92345faedf91b471 [id=48089fc1034648639f16290cb405456a]) successfully deleted
I20250629 01:51:51.313086 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e83009c1599a43d18d1c73c0101ab0be (table default.Table_6b7f597bfe3947f884cf5e129049c99b [id=e4518081a6904de8a019b6d2d015d7b7]) successfully deleted
I20250629 01:51:51.314268 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2540a84a922949e3adb90d0cded1c951
I20250629 01:51:51.314527 1156 ts_tablet_manager.cc:1918] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.314612 1156 log.cc:1199] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e82b1352bb4d49769808dc402c85890f
I20250629 01:51:51.314616 447 catalog_manager.cc:6652] ReplaceTablet: tablet 2540a84a922949e3adb90d0cded1c951 deleted and replaced by tablet df4c253b0c1d41259686c43e202a615f
I20250629 01:51:51.314843 1156 ts_tablet_manager.cc:1939] T e82b1352bb4d49769808dc402c85890f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.316248 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 2540a84a922949e3adb90d0cded1c951 with delete_type TABLET_DATA_DELETED (replaced by tablet df4c253b0c1d41259686c43e202a615f) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.316401 1155 tablet_bootstrap.cc:492] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.317096 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 2540a84a922949e3adb90d0cded1c951 with delete_type TABLET_DATA_DELETED (replaced by tablet df4c253b0c1d41259686c43e202a615f) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.317152 1154 tablet_bootstrap.cc:492] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.317348 1154 ts_tablet_manager.cc:1397] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250629 01:51:51.317557 1155 tablet_bootstrap.cc:654] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.317559 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 2540a84a922949e3adb90d0cded1c951 with delete_type TABLET_DATA_DELETED (replaced by tablet df4c253b0c1d41259686c43e202a615f) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.318049 1159 tablet_bootstrap.cc:492] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.318030 1154 raft_consensus.cc:357] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.318148 1159 ts_tablet_manager.cc:1397] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.154s user 0.015s sys 0.004s
I20250629 01:51:51.318171 1154 raft_consensus.cc:383] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.318225 1154 raft_consensus.cc:738] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.318423 1154 consensus_queue.cc:260] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.320140 1159 raft_consensus.cc:357] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.320279 1159 raft_consensus.cc:383] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.321801 1154 ts_tablet_manager.cc:1428] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:51.322078 1154 tablet_bootstrap.cc:492] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.323892 1158 ts_tablet_manager.cc:1918] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.325443 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5b4701123dfa467ebb6c95f9742b3874
W20250629 01:51:51.326157 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 consensus state has no leader
I20250629 01:51:51.327214 1156 tablet_replica.cc:331] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.327528 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7fdabc198d4b48ebb82cd8a889f7dedf
I20250629 01:51:51.320338 1159 raft_consensus.cc:738] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.328197 1159 consensus_queue.cc:260] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.328691 1159 ts_tablet_manager.cc:1428] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:51.328974 1156 raft_consensus.cc:2241] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.329093 1156 raft_consensus.cc:2270] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.329264 1159 tablet_bootstrap.cc:492] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.329723 1156 ts_tablet_manager.cc:1905] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.331445 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_02a4622c84b1479e96719fa4ad9b0d67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.332248 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02a4622c84b1479e96719fa4ad9b0d67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.332769 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1c68a0f197b241f781976418aed57d90" } new_table_name: "default.Table_d7fd94b003704c088b36cb0df05bd7f5" modify_external_catalogs: true
I20250629 01:51:51.334190 1155 tablet_bootstrap.cc:492] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.334358 1155 ts_tablet_manager.cc:1397] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.004s
I20250629 01:51:51.334861 1155 raft_consensus.cc:357] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.335053 1155 raft_consensus.cc:383] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.335152 1155 raft_consensus.cc:738] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.335188 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9624117e97974a47ac2bfe525da6f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.334327 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b4701123dfa467ebb6c95f9742b3874 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.335587 1155 consensus_queue.cc:260] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.335808 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_32c0db573e334edd9afb9efcdff8d7ad" } modify_external_catalogs: true
I20250629 01:51:51.335994 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_32c0db573e334edd9afb9efcdff8d7ad" } modify_external_catalogs: true
I20250629 01:51:51.336557 1851 webserver.cc:469] Webserver started at http://127.0.104.61:43231/ using document root <none> and password file <none>
I20250629 01:51:51.338107 1851 fs_manager.cc:362] Metadata directory not provided
I20250629 01:51:51.338286 1851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250629 01:51:51.338973 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_37e9efc823274429a519ff980c6caebb" } new_table_name: "default.Table_475fe7b17a5c4aac90faf6432e488d87" modify_external_catalogs: true
I20250629 01:51:51.339205 1155 ts_tablet_manager.cc:1428] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:51.340456 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f0ae4c51924370896c624affde6761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.341243 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_65f2522f26a440dca7e88a9df18ee4e0" } new_table_name: "default.Table_ac0cef7ef73646d1b010c08c31cff2ff" modify_external_catalogs: true
I20250629 01:51:51.342722 1155 tablet_bootstrap.cc:492] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.344425 1156 ts_tablet_manager.cc:1918] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.344507 1156 log.cc:1199] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e1fe3880e51345269e852014d0d60bb6
I20250629 01:51:51.344740 1156 ts_tablet_manager.cc:1939] T e1fe3880e51345269e852014d0d60bb6 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.345536 1155 tablet_bootstrap.cc:654] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.331728 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 557ecee0d6e84f77ad10bbdd27859ee3
I20250629 01:51:51.346529 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_285e96b1ff674385850a3476dd050440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.347031 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_285e96b1ff674385850a3476dd050440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.348742 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fdabc198d4b48ebb82cd8a889f7dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.348919 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b4701123dfa467ebb6c95f9742b3874 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.349052 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.349895 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fdabc198d4b48ebb82cd8a889f7dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.349973 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e82b1352bb4d49769808dc402c85890f (table default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]) successfully deleted
I20250629 01:51:51.350144 1156 tablet_replica.cc:331] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.350183 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e1fe3880e51345269e852014d0d60bb6 (table default.Table_cf2dc25aa17a42fb91e6903713ca0d6c [id=4e38782b406e4cd9ae3278d074a1d97e]) successfully deleted
I20250629 01:51:51.350344 1156 raft_consensus.cc:2241] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:51.350379 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 40266441c3684726a58a5029779395fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40266441c3684726a58a5029779395fc already in progress: creating tablet
I20250629 01:51:51.350522 1156 raft_consensus.cc:2270] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.351136 1156 ts_tablet_manager.cc:1905] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.354229 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4bd3cf3371774492beb8e01eec5349fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.354665 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bd3cf3371774492beb8e01eec5349fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:51.355836 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53660707e9204e2989a63b46d007140a consensus state has no leader
I20250629 01:51:51.357784 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.357971 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 69980d5bf37d4001a6196b22c34ccc94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.361411 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3c70f54de70b4d0e94d25239b097f8fa
I20250629 01:51:51.362412 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c70f54de70b4d0e94d25239b097f8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.364701 1156 ts_tablet_manager.cc:1918] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.364858 1156 log.cc:1199] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/543242e4a5064268ae1e5519b569f682
I20250629 01:51:51.365242 1156 ts_tablet_manager.cc:1939] T 543242e4a5064268ae1e5519b569f682 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.326606 1154 tablet_bootstrap.cc:654] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.366734 1158 log.cc:1199] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:51.366930 1915 raft_consensus.cc:491] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.367033 1915 raft_consensus.cc:513] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.333729 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e962a020664b42ea918e5952dc43aade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.367370 1158 ts_tablet_manager.cc:1939] T 81b32bac5a2144d1943285f5f446d53d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.334757 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b4701123dfa467ebb6c95f9742b3874 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.367590 1915 leader_election.cc:290] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.349520 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 7fdabc198d4b48ebb82cd8a889f7dedf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.354400 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:51.368968 850 leader_election.cc:343] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:51.369197 1158 tablet_replica.cc:331] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:51.369360 850 leader_election.cc:343] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:51.369464 850 leader_election.cc:304] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.369820 1915 raft_consensus.cc:2747] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.369771 1158 raft_consensus.cc:2241] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.372809 1157 ts_tablet_manager.cc:1918] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.372952 1159 tablet_bootstrap.cc:654] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.372954 1157 log.cc:1199] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8a60215417c34bedafbf12172eab469a
I20250629 01:51:51.373260 1157 ts_tablet_manager.cc:1939] T 8a60215417c34bedafbf12172eab469a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.373512 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 717) took 1474 ms. Trace:
I20250629 01:51:51.373572 1157 rpcz_store.cc:276] 0629 01:51:49.899476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.902521 (+ 3045us) service_pool.cc:225] Handling call
0629 01:51:51.256966 (+1354445us) ts_tablet_manager.cc:1145] Deleting tablet 8a60215417c34bedafbf12172eab469a
0629 01:51:51.305796 (+ 48830us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.349603 (+ 43807us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.372740 (+ 23137us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.373498 (+ 758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1353881}
I20250629 01:51:51.374051 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8a60215417c34bedafbf12172eab469a (table default.Table_d20078e8bc4e4a218577b8bc71fc8d44 [id=9653fd4f85744c72a4e48a1f3e7bb112]) successfully deleted
I20250629 01:51:51.374415 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 718) took 1474 ms. Trace:
I20250629 01:51:51.374526 1157 rpcz_store.cc:276] 0629 01:51:49.899571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:49.903265 (+ 3694us) service_pool.cc:225] Handling call
0629 01:51:51.374323 (+1471058us) ts_tablet_manager.cc:1145] Deleting tablet 43b845c3e7174618bd5c394d7206a634
0629 01:51:51.374331 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.374401 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1470740}
W20250629 01:51:51.374940 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 43b845c3e7174618bd5c394d7206a634 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43b845c3e7174618bd5c394d7206a634 already in progress: creating tablet
I20250629 01:51:51.375264 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 743) took 1241 ms. Trace:
I20250629 01:51:51.375387 1157 rpcz_store.cc:276] 0629 01:51:50.134217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.183805 (+ 49588us) service_pool.cc:225] Handling call
0629 01:51:51.375179 (+1191374us) ts_tablet_manager.cc:1145] Deleting tablet 40592f11f430469f9c8e82eba25b38ed
0629 01:51:51.375187 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.375250 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191237}
W20250629 01:51:51.375777 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 40592f11f430469f9c8e82eba25b38ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40592f11f430469f9c8e82eba25b38ed already in progress: creating tablet
I20250629 01:51:51.376085 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 744) took 1241 ms. Trace:
I20250629 01:51:51.376215 1157 rpcz_store.cc:276] 0629 01:51:50.134310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.183951 (+ 49641us) service_pool.cc:225] Handling call
0629 01:51:51.376006 (+1192055us) ts_tablet_manager.cc:1145] Deleting tablet 476a6cf28db04f67bc23ddfc5cacad87
0629 01:51:51.376013 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.376071 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191954}
W20250629 01:51:51.376621 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 476a6cf28db04f67bc23ddfc5cacad87 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 476a6cf28db04f67bc23ddfc5cacad87 already in progress: creating tablet
I20250629 01:51:51.376924 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 741) took 1242 ms. Trace:
I20250629 01:51:51.377022 1157 rpcz_store.cc:276] 0629 01:51:50.134065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.176413 (+ 42348us) service_pool.cc:225] Handling call
0629 01:51:51.376845 (+1200432us) ts_tablet_manager.cc:1145] Deleting tablet 0ba9eabd06174b87a1485f4921895c5f
0629 01:51:51.376853 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.376910 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184222}
W20250629 01:51:51.377422 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0ba9eabd06174b87a1485f4921895c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ba9eabd06174b87a1485f4921895c5f already in progress: creating tablet
I20250629 01:51:51.377733 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 742) took 1243 ms. Trace:
I20250629 01:51:51.377833 1157 rpcz_store.cc:276] 0629 01:51:50.134154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.180355 (+ 46201us) service_pool.cc:225] Handling call
0629 01:51:51.377653 (+1197298us) ts_tablet_manager.cc:1145] Deleting tablet 51b8e24f87b34f15b45270d4693dc7b5
0629 01:51:51.377661 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.377718 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183749}
W20250629 01:51:51.378218 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 51b8e24f87b34f15b45270d4693dc7b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b8e24f87b34f15b45270d4693dc7b5 already in progress: creating tablet
I20250629 01:51:51.378538 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 745) took 1037 ms. Trace:
I20250629 01:51:51.378643 1157 rpcz_store.cc:276] 0629 01:51:50.341221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.341527 (+ 306us) service_pool.cc:225] Handling call
0629 01:51:51.378458 (+1036931us) ts_tablet_manager.cc:1145] Deleting tablet 26d85fd3d3dd4b07a812c019e1a5c748
0629 01:51:51.378466 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.378523 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":969503}
W20250629 01:51:51.379036 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 26d85fd3d3dd4b07a812c019e1a5c748 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 26d85fd3d3dd4b07a812c019e1a5c748 already in progress: creating tablet
I20250629 01:51:51.379366 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 746) took 1038 ms. Trace:
I20250629 01:51:51.379467 1157 rpcz_store.cc:276] 0629 01:51:50.341298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.341567 (+ 269us) service_pool.cc:225] Handling call
0629 01:51:51.379260 (+1037693us) ts_tablet_manager.cc:1145] Deleting tablet c40a07f50971493cb353ed94e850719a
0629 01:51:51.379267 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.379351 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":954616}
I20250629 01:51:51.381068 1157 tablet_replica.cc:331] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.381273 1157 raft_consensus.cc:2241] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.381620 1157 raft_consensus.cc:2270] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.382223 1157 ts_tablet_manager.cc:1905] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.382469 1156 tablet_replica.cc:331] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.382694 1156 raft_consensus.cc:2241] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.382853 1156 raft_consensus.cc:2270] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.383526 1156 ts_tablet_manager.cc:1905] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.380646 1851 fs_manager.cc:714] Time spent opening directory manager: real 0.033s user 0.003s sys 0.006s
I20250629 01:51:51.387748 1159 tablet_bootstrap.cc:492] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.387898 1159 ts_tablet_manager.cc:1397] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.010s
I20250629 01:51:51.388635 1159 raft_consensus.cc:357] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.388911 1159 raft_consensus.cc:383] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.388998 1159 raft_consensus.cc:738] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.389212 1159 consensus_queue.cc:260] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.389757 1159 ts_tablet_manager.cc:1428] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:51.389897 1156 ts_tablet_manager.cc:1918] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.390012 1156 log.cc:1199] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9624117e97974a47ac2bfe525da6f405
I20250629 01:51:51.390823 1156 ts_tablet_manager.cc:1939] T 9624117e97974a47ac2bfe525da6f405 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.391505 1156 tablet_replica.cc:331] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.391654 1159 tablet_bootstrap.cc:492] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.391690 1156 raft_consensus.cc:2241] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.392138 1156 raft_consensus.cc:2270] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.392719 1156 ts_tablet_manager.cc:1905] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.393108 1159 tablet_bootstrap.cc:654] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.397286 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 0a574f95ac2a4193a563e63cbe120768 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.398216 1156 ts_tablet_manager.cc:1918] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:51.398386 1156 log.cc:1199] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5b4701123dfa467ebb6c95f9742b3874
I20250629 01:51:51.398787 1156 ts_tablet_manager.cc:1939] T 5b4701123dfa467ebb6c95f9742b3874 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.402426 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f2c10fb36e7d4a3db070958ca3ffeeb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.402892 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2c10fb36e7d4a3db070958ca3ffeeb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.404060 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c70f54de70b4d0e94d25239b097f8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.404664 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 81b32bac5a2144d1943285f5f446d53d (table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]) successfully deleted
I20250629 01:51:51.404840 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 543242e4a5064268ae1e5519b569f682 (table default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]) successfully deleted
W20250629 01:51:51.405320 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5ab0aa4e74d345b2a214b90b5018c82b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ab0aa4e74d345b2a214b90b5018c82b already in progress: creating tablet
W20250629 01:51:51.405522 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 845f2422d6494c95a4738f3062c216c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 845f2422d6494c95a4738f3062c216c7 already in progress: creating tablet
W20250629 01:51:51.405711 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5c1d273c70ac4db7a215f02ecac11154 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c1d273c70ac4db7a215f02ecac11154 already in progress: creating tablet
I20250629 01:51:51.405802 1159 tablet_bootstrap.cc:492] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.405893 1159 ts_tablet_manager.cc:1397] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20250629 01:51:51.405854 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7dde0c66922743c1b227bc48f1adbd55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.406239 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dde0c66922743c1b227bc48f1adbd55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.406387 1159 raft_consensus.cc:357] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.406500 1159 raft_consensus.cc:383] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.406536 1159 raft_consensus.cc:738] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.406710 1159 consensus_queue.cc:260] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:51.405884 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7eb02d3ff1104f6885ea1736852bbe57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eb02d3ff1104f6885ea1736852bbe57 already in progress: creating tablet
W20250629 01:51:51.407017 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 already in progress: creating tablet
W20250629 01:51:51.407191 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9bbbeb43473a4f6f892056a30f62589f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bbbeb43473a4f6f892056a30f62589f already in progress: creating tablet
W20250629 01:51:51.407449 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2540a84a922949e3adb90d0cded1c951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2540a84a922949e3adb90d0cded1c951 already in progress: creating tablet
I20250629 01:51:51.407613 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9624117e97974a47ac2bfe525da6f405 (table default.Table_3906fb981a8640e1906a78a321c479b2 [id=e2a731484fe346738ed611d5d7312092]) successfully deleted
I20250629 01:51:51.408469 1964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250629 01:51:51.409477 1159 ts_tablet_manager.cc:1428] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20250629 01:51:51.410342 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 347fa6a12a744ee88058643b7143e39d consensus state has no leader
I20250629 01:51:51.410645 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b1a6331211334c548b6c97e62280bdff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.410959 1851 fs_manager.cc:730] Time spent opening block manager: real 0.026s user 0.001s sys 0.003s
W20250629 01:51:51.410981 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1a6331211334c548b6c97e62280bdff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.411897 1159 tablet_bootstrap.cc:492] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:51.411958 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bf612fea5b74a0bb78700cc07c4d68d consensus state has no leader
I20250629 01:51:51.412254 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e36a71de95204b6fbf931a4ce70fb501" } new_table_name: "default.Table_648891736c08404eb59b4792a5970064" modify_external_catalogs: true
I20250629 01:51:51.412581 1156 tablet_replica.cc:331] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.412799 1156 raft_consensus.cc:2241] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.412906 1156 raft_consensus.cc:2270] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.413172 1159 tablet_bootstrap.cc:654] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.413506 1156 ts_tablet_manager.cc:1905] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.413758 1157 ts_tablet_manager.cc:1918] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.413839 1157 log.cc:1199] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4dbe70d4e4e74c6ba7db2766bd78c8ad
I20250629 01:51:51.414077 1157 ts_tablet_manager.cc:1939] T 4dbe70d4e4e74c6ba7db2766bd78c8ad P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.414323 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 757) took 1011 ms. Trace:
I20250629 01:51:51.411108 1851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data,/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
uuid: "0f4258c004e04bcf8216bb7c8fd56c54"
format_stamp: "Formatted at 2025-06-29 01:51:36 on dist-test-slave-0pfc"
I20250629 01:51:51.415323 1157 rpcz_store.cc:276] 0629 01:51:50.402988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.404413 (+ 1425us) service_pool.cc:225] Handling call
0629 01:51:51.381034 (+976621us) ts_tablet_manager.cc:1145] Deleting tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad
0629 01:51:51.381041 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.387995 (+ 6954us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.401085 (+ 13090us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.414312 (+ 13227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":929967}
I20250629 01:51:51.415724 1851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250629 01:51:51.416281 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4dbe70d4e4e74c6ba7db2766bd78c8ad (table default.Table_adc0a47bc8504a73aa8b9ede02097dd9 [id=604220139d4f4b31bfaca78aa2b32f29]) successfully deleted
I20250629 01:51:51.416700 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 758) took 1013 ms. Trace:
I20250629 01:51:51.370090 1158 raft_consensus.cc:2270] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.422748 1158 ts_tablet_manager.cc:1905] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.390612 1154 tablet_bootstrap.cc:492] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.404331 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c70f54de70b4d0e94d25239b097f8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.423540 1154 ts_tablet_manager.cc:1397] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.102s user 0.001s sys 0.010s
I20250629 01:51:51.423121 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ed143377709465d9ae5df7c2ec3ca47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:41 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.423895 1157 rpcz_store.cc:276] 0629 01:51:50.403078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.404553 (+ 1475us) service_pool.cc:225] Handling call
0629 01:51:51.416582 (+1012029us) ts_tablet_manager.cc:1145] Deleting tablet 44ab886d05e0468ab0e7b156a32dd392
0629 01:51:51.416590 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.416687 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":965467}
W20250629 01:51:51.424264 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 44ab886d05e0468ab0e7b156a32dd392 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ab886d05e0468ab0e7b156a32dd392 already in progress: creating tablet
I20250629 01:51:51.424266 1154 raft_consensus.cc:357] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.424409 1154 raft_consensus.cc:383] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.424465 1154 raft_consensus.cc:738] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.424672 1154 consensus_queue.cc:260] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:51.424933 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4c7b76cb56b24604b13ee29c7f576ce2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c7b76cb56b24604b13ee29c7f576ce2 already in progress: creating tablet
I20250629 01:51:51.425103 1154 ts_tablet_manager.cc:1428] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:51.425351 1157 tablet_replica.cc:331] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.425374 1154 tablet_bootstrap.cc:492] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.425442 1156 ts_tablet_manager.cc:1918] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.425516 1156 log.cc:1199] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/69f0ae4c51924370896c624affde6761
I20250629 01:51:51.425740 1156 ts_tablet_manager.cc:1939] T 69f0ae4c51924370896c624affde6761 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.426059 1157 raft_consensus.cc:2241] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.426183 1157 raft_consensus.cc:2270] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.426591 1157 ts_tablet_manager.cc:1905] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.426776 1154 tablet_bootstrap.cc:654] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.428987 1158 ts_tablet_manager.cc:1918] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.429090 1158 log.cc:1199] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/683c87ea2ad44cecb2366d4a846d9195
I20250629 01:51:51.429352 1158 ts_tablet_manager.cc:1939] T 683c87ea2ad44cecb2366d4a846d9195 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.431644 1157 ts_tablet_manager.cc:1918] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.442399 1157 log.cc:1199] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b7c7f2ac44734a39ab6434a74c91b9e4
I20250629 01:51:51.442832 1157 ts_tablet_manager.cc:1939] T b7c7f2ac44734a39ab6434a74c91b9e4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.443609 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b7c7f2ac44734a39ab6434a74c91b9e4 (table default.Table_fa24ba8423a34245b118894025a8b4f4 [id=a07cbcbb14984c538bdd7d1441d52055]) successfully deleted
I20250629 01:51:51.444299 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 759) took 1003 ms. Trace:
I20250629 01:51:51.444665 1157 rpcz_store.cc:276] 0629 01:51:50.440446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.467000 (+ 26554us) service_pool.cc:225] Handling call
0629 01:51:51.444177 (+977177us) ts_tablet_manager.cc:1145] Deleting tablet 9b54228f0013424588a355ceb17d9784
0629 01:51:51.444185 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.444269 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977039}
W20250629 01:51:51.445405 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9b54228f0013424588a355ceb17d9784 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9b54228f0013424588a355ceb17d9784 already in progress: creating tablet
W20250629 01:51:51.446070 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 86099eb9e40a4ea389afc059f90a97ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86099eb9e40a4ea389afc059f90a97ef already in progress: creating tablet
I20250629 01:51:51.446512 1154 tablet_bootstrap.cc:492] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.446693 1154 ts_tablet_manager.cc:1397] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
W20250629 01:51:51.446956 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5a3752f8a275400da8cfff249f2dc6ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a3752f8a275400da8cfff249f2dc6ee already in progress: creating tablet
I20250629 01:51:51.447440 1157 tablet_replica.cc:331] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.447443 1154 raft_consensus.cc:357] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.447660 1154 raft_consensus.cc:383] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.449013 1155 tablet_bootstrap.cc:492] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.449182 1155 ts_tablet_manager.cc:1397] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.014s
I20250629 01:51:51.449913 1155 raft_consensus.cc:357] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.450119 1155 raft_consensus.cc:383] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.450349 1155 raft_consensus.cc:738] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.450628 1155 consensus_queue.cc:260] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.450960 449 master_service.cc:948] ReplaceTablet: received request to replace tablet a20245e23a9e4dff9bfb1e565dc38184 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:51.453459 1155 ts_tablet_manager.cc:1428] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:51.453769 1155 tablet_bootstrap.cc:492] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.455356 1155 tablet_bootstrap.cc:654] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.455343 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.455672 1157 raft_consensus.cc:2241] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.455694 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 683c87ea2ad44cecb2366d4a846d9195 (table default.Table_04cc9d29a25645f0b3facf31bd7a9f57 [id=395f2b984ff9443a90e78e8d44bca991]) successfully deleted
I20250629 01:51:51.455873 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
I20250629 01:51:51.455896 1157 raft_consensus.cc:2270] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.456077 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 69f0ae4c51924370896c624affde6761 (table default.Table_becfa32b77bb40b1b9df61126ffe628e [id=1c654e447e1b46f1a4570b41df8b1861]) successfully deleted
W20250629 01:51:51.456287 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7fdabc198d4b48ebb82cd8a889f7dedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fdabc198d4b48ebb82cd8a889f7dedf already in progress: creating tablet
W20250629 01:51:51.456481 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 557ecee0d6e84f77ad10bbdd27859ee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 557ecee0d6e84f77ad10bbdd27859ee3 already in progress: creating tablet
W20250629 01:51:51.456700 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3c70f54de70b4d0e94d25239b097f8fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c70f54de70b4d0e94d25239b097f8fa already in progress: creating tablet
I20250629 01:51:51.458195 1157 ts_tablet_manager.cc:1905] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.460232 1159 tablet_bootstrap.cc:492] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.460391 1159 ts_tablet_manager.cc:1397] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.007s
I20250629 01:51:51.461194 1159 raft_consensus.cc:357] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.461388 1159 raft_consensus.cc:383] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.461468 1159 raft_consensus.cc:738] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.461709 1159 consensus_queue.cc:260] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.463989 1159 ts_tablet_manager.cc:1428] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:51.465191 1159 tablet_bootstrap.cc:492] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:51.465910 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2045012c80d049eaabf7dec1df28b087 consensus state has no leader
I20250629 01:51:51.467550 1155 tablet_bootstrap.cc:492] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.469039 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 26d85fd3d3dd4b07a812c019e1a5c748 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.469728 1159 tablet_bootstrap.cc:654] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.470613 1155 ts_tablet_manager.cc:1397] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250629 01:51:51.471211 1155 raft_consensus.cc:357] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.471603 1155 raft_consensus.cc:383] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.471693 1155 raft_consensus.cc:738] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.471931 1155 consensus_queue.cc:260] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.474110 1155 ts_tablet_manager.cc:1428] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:51.474725 1155 tablet_bootstrap.cc:492] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.476647 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4db6832e616444be90ea5b21d5dace4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.478274 1155 tablet_bootstrap.cc:654] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.480636 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9396b8200cc74c71b35de340567589a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.484812 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9396b8200cc74c71b35de340567589a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:51.485694 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4db6832e616444be90ea5b21d5dace4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.487200 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_14435d8c8f2846e393084f1fa1f7f303"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.487936 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14435d8c8f2846e393084f1fa1f7f303 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.489987 1157 ts_tablet_manager.cc:1918] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.496030 1157 log.cc:1199] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0b7a640451934bff8672cd88dc86c529
I20250629 01:51:51.496424 1157 ts_tablet_manager.cc:1939] T 0b7a640451934bff8672cd88dc86c529 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.496939 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0b7a640451934bff8672cd88dc86c529 (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]) successfully deleted
I20250629 01:51:51.497220 1157 tablet_replica.cc:331] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.497411 1157 raft_consensus.cc:2241] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.497610 1157 raft_consensus.cc:2270] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.498117 1157 ts_tablet_manager.cc:1905] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.498617 1159 tablet_bootstrap.cc:492] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.498764 1159 ts_tablet_manager.cc:1397] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.002s
I20250629 01:51:51.499405 1159 raft_consensus.cc:357] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.499605 1159 raft_consensus.cc:383] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.499686 1159 raft_consensus.cc:738] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.499933 1159 consensus_queue.cc:260] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.500205 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 7908afa2abaa4521b44f954e0790f986 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:51.500612 1159 ts_tablet_manager.cc:1428] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:51.500964 1159 tablet_bootstrap.cc:492] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.502657 1159 tablet_bootstrap.cc:654] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.507510 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a32636c293bc44858da7f29a515c467b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.510284 1154 raft_consensus.cc:738] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.511943 1154 consensus_queue.cc:260] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.515846 1824 raft_consensus.cc:491] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.515976 1824 raft_consensus.cc:513] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.516235 1851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250629 01:51:51.516501 1824 leader_election.cc:290] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.517220 1851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250629 01:51:51.517585 1851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250629 01:51:51.520272 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "798516e55ca747fcb2942026adb78af0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.520895 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "798516e55ca747fcb2942026adb78af0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:51.521057 1048 raft_consensus.cc:2466] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:51:51.521109 712 leader_election.cc:343] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.521472 710 leader_election.cc:304] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:51.521909 1824 raft_consensus.cc:2802] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:51.521973 1824 raft_consensus.cc:491] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:51.522022 1824 raft_consensus.cc:3058] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.523211 1157 ts_tablet_manager.cc:1918] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:51.523299 1157 log.cc:1199] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e508393b9cb044c4b90194c8aab3184a
I20250629 01:51:51.524372 1824 raft_consensus.cc:513] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.525197 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "798516e55ca747fcb2942026adb78af0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:51.525622 712 leader_election.cc:343] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.526163 1824 leader_election.cc:290] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.526219 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "798516e55ca747fcb2942026adb78af0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:51.526348 1048 raft_consensus.cc:3058] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.526407 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 352a8dbd7f164ca9b622fed30bc6beeb
I20250629 01:51:51.527261 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 352a8dbd7f164ca9b622fed30bc6beeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.528620 1048 raft_consensus.cc:2466] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:51.529033 710 leader_election.cc:304] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:51.529445 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 352a8dbd7f164ca9b622fed30bc6beeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.529718 1824 raft_consensus.cc:2802] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:51.529858 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_02a4622c84b1479e96719fa4ad9b0d67" } modify_external_catalogs: true
I20250629 01:51:51.529994 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_02a4622c84b1479e96719fa4ad9b0d67" } modify_external_catalogs: true
I20250629 01:51:51.530068 1824 raft_consensus.cc:695] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:51.530259 1824 consensus_queue.cc:237] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.531183 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5407d7aa15024faab9edbf55a1485511" } new_table_name: "default.Table_09dc040be7aa44f18e37086a158aad84" modify_external_catalogs: true
I20250629 01:51:51.532831 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 352a8dbd7f164ca9b622fed30bc6beeb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:51.533470 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 352a8dbd7f164ca9b622fed30bc6beeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352a8dbd7f164ca9b622fed30bc6beeb already in progress: creating tablet
I20250629 01:51:51.534365 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7908afa2abaa4521b44f954e0790f986
I20250629 01:51:51.534745 447 catalog_manager.cc:6652] ReplaceTablet: tablet 7908afa2abaa4521b44f954e0790f986 deleted and replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8
I20250629 01:51:51.535638 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 7908afa2abaa4521b44f954e0790f986 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.536477 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1f284b2051314392a75e40bc77ef6a3f" } modify_external_catalogs: true
I20250629 01:51:51.536525 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_899f5cb3f36f46dd8e4c809803070720"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.536774 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1f284b2051314392a75e40bc77ef6a3f" } modify_external_catalogs: true
W20250629 01:51:51.536995 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_899f5cb3f36f46dd8e4c809803070720 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.539340 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 7908afa2abaa4521b44f954e0790f986 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.539407 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7908afa2abaa4521b44f954e0790f986 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.540012 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
I20250629 01:51:51.540922 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a3586fcfc73547d2acf71a9fae315145
W20250629 01:51:51.541100 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7908afa2abaa4521b44f954e0790f986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7908afa2abaa4521b44f954e0790f986 already in progress: creating tablet
W20250629 01:51:51.541970 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dee14f26866c48cfbf66ee83c8d7ccd4 has no consensus state
I20250629 01:51:51.542085 454 catalog_manager.cc:5582] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:51.542418 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:51.543076 890 tablet_service.cc:1468] Processing CreateTablet for tablet 83e0f20f27264ff5b064f14638029f79 (DEFAULT_TABLE table=default.Table_e50b86b187294b3fb9f068be98e6101d [id=7c6f6ae5bb1d4467acec8b7fbe012e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.543792 1915 raft_consensus.cc:491] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.543896 1915 raft_consensus.cc:513] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.544356 1915 leader_election.cc:290] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.544549 888 tablet_service.cc:1468] Processing CreateTablet for tablet 9e62e0683ffd4fdb80cd9646ddcc6692 (DEFAULT_TABLE table=default.Table_33af2617f1ad46a3b9a25fe7c34f4df6 [id=740ed2bcafd142f6a6e482ae28302d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.545009 1850 consensus_queue.cc:1035] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:51:51.545173 850 leader_election.cc:343] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0c712d978a3848b9bb3abdbce8314dec
W20250629 01:51:51.545476 850 leader_election.cc:343] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0c712d978a3848b9bb3abdbce8314dec
I20250629 01:51:51.545557 850 leader_election.cc:304] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.545969 1915 raft_consensus.cc:2747] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.546090 896 tablet_service.cc:1468] Processing CreateTablet for tablet e332a512574e4b4e881aeb37e289f0bc (DEFAULT_TABLE table=default.Table_8e2c9df275d2453daa8b73a723d56ed6 [id=8135eb94530a4640b5e3f8f3081f50d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.546669 881 tablet_service.cc:1468] Processing CreateTablet for tablet 3d932c3240624c07ab5c43f70112f326 (DEFAULT_TABLE table=default.Table_9d377064eeb847699a17f707f0cae83c [id=36da40bf1ad24acda455c88733bfdb90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.547370 877 tablet_service.cc:1468] Processing CreateTablet for tablet 10920d584d0c456d8a5eb8efb81da36f (DEFAULT_TABLE table=default.Table_540c577da2d044a19efbc470511fcf47 [id=71d875cddda04540ac264a8fb4842d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.548303 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ab9e746444a24266aca5439c51b30a27" } modify_external_catalogs: true
I20250629 01:51:51.548465 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ab9e746444a24266aca5439c51b30a27" } modify_external_catalogs: true
I20250629 01:51:51.549353 1923 raft_consensus.cc:491] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.549489 1923 raft_consensus.cc:513] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.550097 1923 leader_election.cc:290] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.550908 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "241a5c9aee3f4297b67b18380ef5ef88" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:51.551116 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "241a5c9aee3f4297b67b18380ef5ef88" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.551501 1158 tablet_replica.cc:331] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:51.551774 986 leader_election.cc:343] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.551844 1158 raft_consensus.cc:2241] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.552043 1158 raft_consensus.cc:2270] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:51.552249 988 leader_election.cc:343] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.552366 988 leader_election.cc:304] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.552824 1915 raft_consensus.cc:491] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.552953 1915 raft_consensus.cc:513] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.553483 1915 leader_election.cc:290] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.553699 1923 raft_consensus.cc:2747] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:51.554778 850 leader_election.cc:343] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a32636c293bc44858da7f29a515c467b
I20250629 01:51:51.553085 1158 ts_tablet_manager.cc:1905] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:51.555106 850 leader_election.cc:343] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: a32636c293bc44858da7f29a515c467b
I20250629 01:51:51.555187 850 leader_election.cc:304] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.555594 1915 raft_consensus.cc:2747] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.557009 1155 tablet_bootstrap.cc:492] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.557049 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9a5f762a18104801991847cafb8e0e97" } new_table_name: "default.Table_f798e82e14704a689fe84cbeb9a8d477" modify_external_catalogs: true
I20250629 01:51:51.557266 1155 ts_tablet_manager.cc:1397] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.006s
I20250629 01:51:51.558511 1154 ts_tablet_manager.cc:1428] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.112s user 0.015s sys 0.003s
I20250629 01:51:51.559120 1155 raft_consensus.cc:357] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.559398 1155 raft_consensus.cc:383] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.559525 1155 raft_consensus.cc:738] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.559943 1155 consensus_queue.cc:260] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.560889 1155 ts_tablet_manager.cc:1428] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:51.561897 1850 raft_consensus.cc:491] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.561993 1850 raft_consensus.cc:513] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.562386 1850 leader_election.cc:290] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.562870 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:51.563005 1049 raft_consensus.cc:2466] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:51.563334 710 leader_election.cc:304] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:51.563539 1850 raft_consensus.cc:2802] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:51.563599 1850 raft_consensus.cc:491] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:51.563645 1850 raft_consensus.cc:3058] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.563625 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.563803 1155 tablet_bootstrap.cc:492] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:51.565362 712 leader_election.cc:343] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.565954 1850 raft_consensus.cc:513] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.566138 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d532629a716e44368b4324cf8b41f873" } modify_external_catalogs: true
I20250629 01:51:51.566256 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d532629a716e44368b4324cf8b41f873" } modify_external_catalogs: true
I20250629 01:51:51.566532 1850 leader_election.cc:290] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.567224 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:51.567368 1049 raft_consensus.cc:3058] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.567768 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:51.568261 712 leader_election.cc:343] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.569166 738 tablet_service.cc:1468] Processing CreateTablet for tablet 83e0f20f27264ff5b064f14638029f79 (DEFAULT_TABLE table=default.Table_e50b86b187294b3fb9f068be98e6101d [id=7c6f6ae5bb1d4467acec8b7fbe012e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.569561 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e0f20f27264ff5b064f14638029f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.569682 1049 raft_consensus.cc:2466] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:51.571249 710 leader_election.cc:304] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:51.562557 1154 tablet_bootstrap.cc:492] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.574051 1155 tablet_bootstrap.cc:654] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.575700 1824 raft_consensus.cc:2802] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:51.575910 1824 raft_consensus.cc:695] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:51.579236 1824 consensus_queue.cc:237] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.580020 878 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae0d61d254c4e1ba508e30aa35600ca (DEFAULT_TABLE table=default.Table_9abd434ca0b940478c9661546ed480a2 [id=68eb8cc1cacd46cf8bd81fb871c5d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.581826 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 83e0f20f27264ff5b064f14638029f79 (DEFAULT_TABLE table=default.Table_e50b86b187294b3fb9f068be98e6101d [id=7c6f6ae5bb1d4467acec8b7fbe012e18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.582144 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e0f20f27264ff5b064f14638029f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.582126 738 tablet_service.cc:1468] Processing CreateTablet for tablet 9e62e0683ffd4fdb80cd9646ddcc6692 (DEFAULT_TABLE table=default.Table_33af2617f1ad46a3b9a25fe7c34f4df6 [id=740ed2bcafd142f6a6e482ae28302d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.582690 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e62e0683ffd4fdb80cd9646ddcc6692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.588270 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 9e62e0683ffd4fdb80cd9646ddcc6692 (DEFAULT_TABLE table=default.Table_33af2617f1ad46a3b9a25fe7c34f4df6 [id=740ed2bcafd142f6a6e482ae28302d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.588935 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e62e0683ffd4fdb80cd9646ddcc6692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.589171 743 tablet_service.cc:1468] Processing CreateTablet for tablet e332a512574e4b4e881aeb37e289f0bc (DEFAULT_TABLE table=default.Table_8e2c9df275d2453daa8b73a723d56ed6 [id=8135eb94530a4640b5e3f8f3081f50d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.589567 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e332a512574e4b4e881aeb37e289f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:51.597445 712 consensus_peers.cc:489] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:51.599265 1029 tablet_service.cc:1468] Processing CreateTablet for tablet e332a512574e4b4e881aeb37e289f0bc (DEFAULT_TABLE table=default.Table_8e2c9df275d2453daa8b73a723d56ed6 [id=8135eb94530a4640b5e3f8f3081f50d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.599562 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e332a512574e4b4e881aeb37e289f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.599995 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3d932c3240624c07ab5c43f70112f326 (DEFAULT_TABLE table=default.Table_9d377064eeb847699a17f707f0cae83c [id=36da40bf1ad24acda455c88733bfdb90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.600533 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d932c3240624c07ab5c43f70112f326. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.606701 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3d932c3240624c07ab5c43f70112f326 (DEFAULT_TABLE table=default.Table_9d377064eeb847699a17f707f0cae83c [id=36da40bf1ad24acda455c88733bfdb90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.606966 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d932c3240624c07ab5c43f70112f326. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.607517 755 tablet_service.cc:1468] Processing CreateTablet for tablet 10920d584d0c456d8a5eb8efb81da36f (DEFAULT_TABLE table=default.Table_540c577da2d044a19efbc470511fcf47 [id=71d875cddda04540ac264a8fb4842d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.607533 895 tablet_service.cc:1468] Processing CreateTablet for tablet 290c5ab8364145a39f6ff403ab7b20b1 (DEFAULT_TABLE table=default.Table_20d462c5b8284f99b445c7539d756d3b [id=8705c82c65df415abf3299b10b90f48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.607651 879 tablet_service.cc:1468] Processing CreateTablet for tablet df0fe7f5469d44fc8f6ac5c72377b48a (DEFAULT_TABLE table=default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.607795 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10920d584d0c456d8a5eb8efb81da36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.610250 889 tablet_service.cc:1468] Processing CreateTablet for tablet a0002dca4db24405afdce3e1076eb52e (DEFAULT_TABLE table=default.Table_46cfbca358ce43a99a1a1dc43600e2b1 [id=91daa3f10462457ea1270f4f7bf7257d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.614063 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83e0f20f27264ff5b064f14638029f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.614217 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae0d61d254c4e1ba508e30aa35600ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.615127 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_93f95da30d034c31b5c7f2c52229fcf2" } new_table_name: "default.Table_a5007857174542ea9967e05f32de9d93" modify_external_catalogs: true
I20250629 01:51:51.616181 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290c5ab8364145a39f6ff403ab7b20b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.616281 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2c656b7d46064f7d811de6d03aa4224e" } new_table_name: "default.Table_884f76ae76564136b99617ba1c1ca0a8" modify_external_catalogs: true
I20250629 01:51:51.618379 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e62e0683ffd4fdb80cd9646ddcc6692. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.619269 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9ffe7a3207f44327bb5b618169fa2aaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.619707 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ffe7a3207f44327bb5b618169fa2aaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.620241 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0002dca4db24405afdce3e1076eb52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.620633 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0fe7f5469d44fc8f6ac5c72377b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.622582 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e332a512574e4b4e881aeb37e289f0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.624779 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d932c3240624c07ab5c43f70112f326. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.626104 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10920d584d0c456d8a5eb8efb81da36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.639111 1157 ts_tablet_manager.cc:1939] T e508393b9cb044c4b90194c8aab3184a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.640591 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 764) took 1034 ms. Trace:
I20250629 01:51:51.641304 1157 rpcz_store.cc:276] 0629 01:51:50.605760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.609642 (+ 3882us) service_pool.cc:225] Handling call
0629 01:51:51.497181 (+887539us) ts_tablet_manager.cc:1145] Deleting tablet e508393b9cb044c4b90194c8aab3184a
0629 01:51:51.497189 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.505198 (+ 8009us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.521064 (+ 15866us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:51.640578 (+119514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":887389}
W20250629 01:51:51.642091 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 already in progress: creating tablet
I20250629 01:51:51.642215 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e508393b9cb044c4b90194c8aab3184a (table default.Table_c19ff63b162e4df9b84be3f39a8ae63f [id=c6883d7c8bd241f78a143cda44abfda6]) successfully deleted
W20250629 01:51:51.642288 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9bbbeb43473a4f6f892056a30f62589f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bbbeb43473a4f6f892056a30f62589f already in progress: creating tablet
I20250629 01:51:51.642730 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 766) took 1008 ms. Trace:
I20250629 01:51:51.643042 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 10920d584d0c456d8a5eb8efb81da36f (DEFAULT_TABLE table=default.Table_540c577da2d044a19efbc470511fcf47 [id=71d875cddda04540ac264a8fb4842d8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.643353 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10920d584d0c456d8a5eb8efb81da36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.643358 1157 rpcz_store.cc:276] 0629 01:51:50.633841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:50.633885 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:51.642626 (+1008741us) ts_tablet_manager.cc:1145] Deleting tablet 2aca26af31ed4bc994f4d62603edc881
0629 01:51:51.642636 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:51.642715 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008595}
W20250629 01:51:51.643720 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2aca26af31ed4bc994f4d62603edc881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2aca26af31ed4bc994f4d62603edc881 already in progress: creating tablet
I20250629 01:51:51.644351 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7039f5dc37154a69870a9955aa07d568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.644577 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 39d2adf801934179825f96a5f7b3c303 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39d2adf801934179825f96a5f7b3c303 already in progress: creating tablet
I20250629 01:51:51.644544 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae0d61d254c4e1ba508e30aa35600ca (DEFAULT_TABLE table=default.Table_9abd434ca0b940478c9661546ed480a2 [id=68eb8cc1cacd46cf8bd81fb871c5d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.644820 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae0d61d254c4e1ba508e30aa35600ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:51.644835 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7039f5dc37154a69870a9955aa07d568 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:51.645458 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 01d95356c7a7419bb56ddf67d451eadc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01d95356c7a7419bb56ddf67d451eadc already in progress: creating tablet
I20250629 01:51:51.646023 1157 tablet_replica.cc:331] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.646167 1157 raft_consensus.cc:2241] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.647231 1157 raft_consensus.cc:2270] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.647405 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0467b153057f491e81681fa399a2a1a9
I20250629 01:51:51.647794 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae0d61d254c4e1ba508e30aa35600ca (DEFAULT_TABLE table=default.Table_9abd434ca0b940478c9661546ed480a2 [id=68eb8cc1cacd46cf8bd81fb871c5d645]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.648043 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae0d61d254c4e1ba508e30aa35600ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.648267 1157 ts_tablet_manager.cc:1905] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.648579 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 0467b153057f491e81681fa399a2a1a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.655602 750 tablet_service.cc:1468] Processing CreateTablet for tablet 290c5ab8364145a39f6ff403ab7b20b1 (DEFAULT_TABLE table=default.Table_20d462c5b8284f99b445c7539d756d3b [id=8705c82c65df415abf3299b10b90f48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.655867 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290c5ab8364145a39f6ff403ab7b20b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.662297 1157 ts_tablet_manager.cc:1918] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.662402 1157 log.cc:1199] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d61687e19d994e87b09e55bd3b9476d7
I20250629 01:51:51.662389 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:51.662647 1157 ts_tablet_manager.cc:1939] T d61687e19d994e87b09e55bd3b9476d7 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.663060 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d61687e19d994e87b09e55bd3b9476d7 (table default.Table_81c8bc2326604f6bbc1f5a72cf8e10fb [id=f274d40e73944253825a87b39ee56b71]) successfully deleted
I20250629 01:51:51.663295 1850 raft_consensus.cc:491] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.663399 1850 raft_consensus.cc:513] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.663915 1850 leader_election.cc:290] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.664007 1850 raft_consensus.cc:491] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.664768 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f7c1f1af39d47c191d34dc026f7f687" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.664080 1850 raft_consensus.cc:513] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.665885 1850 leader_election.cc:290] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.665885 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
W20250629 01:51:51.665268 710 leader_election.cc:343] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2f7c1f1af39d47c191d34dc026f7f687
W20250629 01:51:51.666461 712 leader_election.cc:343] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.666769 712 leader_election.cc:304] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:51.666810 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2d37334db2b9465a8fcf5e849d5d3b0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d37334db2b9465a8fcf5e849d5d3b0c already in progress: creating tablet
I20250629 01:51:51.666916 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af06514069d14e6caea07fb498630608"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.667387 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af06514069d14e6caea07fb498630608 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.667452 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9b680859a1a4c56afe33f5b1b3c0793" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.667618 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f52937830a924d6d8338d78da24b84cd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f52937830a924d6d8338d78da24b84cd already in progress: creating tablet
W20250629 01:51:51.667975 710 leader_election.cc:343] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b9b680859a1a4c56afe33f5b1b3c0793
W20250629 01:51:51.668220 712 leader_election.cc:343] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.668268 1157 tablet_replica.cc:331] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.668303 712 leader_election.cc:304] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:51.668421 1157 raft_consensus.cc:2241] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.668530 1157 raft_consensus.cc:2270] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.668581 1861 raft_consensus.cc:2747] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.668709 1861 raft_consensus.cc:2747] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.668943 1157 ts_tablet_manager.cc:1905] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.676956 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 290c5ab8364145a39f6ff403ab7b20b1 (DEFAULT_TABLE table=default.Table_20d462c5b8284f99b445c7539d756d3b [id=8705c82c65df415abf3299b10b90f48c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.677383 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290c5ab8364145a39f6ff403ab7b20b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.678295 1157 ts_tablet_manager.cc:1918] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.678398 1157 log.cc:1199] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0c712d978a3848b9bb3abdbce8314dec
I20250629 01:51:51.678644 1157 ts_tablet_manager.cc:1939] T 0c712d978a3848b9bb3abdbce8314dec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.678597 755 tablet_service.cc:1468] Processing CreateTablet for tablet df0fe7f5469d44fc8f6ac5c72377b48a (DEFAULT_TABLE table=default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.678932 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0fe7f5469d44fc8f6ac5c72377b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.678963 1159 tablet_bootstrap.cc:492] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.679080 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0c712d978a3848b9bb3abdbce8314dec (table default.Table_ca2bcac172414e6c9f264cb2202a76fa [id=dfc541fca28746b8ba7e1f81a24c7fce]) successfully deleted
I20250629 01:51:51.679121 1159 ts_tablet_manager.cc:1397] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.178s user 0.004s sys 0.002s
I20250629 01:51:51.679842 1159 raft_consensus.cc:357] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.680006 1159 raft_consensus.cc:383] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.680048 1159 raft_consensus.cc:738] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.680269 1159 consensus_queue.cc:260] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.680769 1159 ts_tablet_manager.cc:1428] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:51.613889 1154 tablet_bootstrap.cc:654] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.682507 1159 tablet_bootstrap.cc:492] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.683663 1159 tablet_bootstrap.cc:654] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.686026 1923 raft_consensus.cc:491] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.686159 1923 raft_consensus.cc:513] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.686792 1923 leader_election.cc:290] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.688632 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e962a020664b42ea918e5952dc43aade" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.689030 988 leader_election.cc:343] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.689401 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e962a020664b42ea918e5952dc43aade" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:51.689667 1155 tablet_bootstrap.cc:492] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
W20250629 01:51:51.689764 986 leader_election.cc:343] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.689792 1155 ts_tablet_manager.cc:1397] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.013s
I20250629 01:51:51.689846 986 leader_election.cc:304] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.690193 1923 raft_consensus.cc:2747] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.690369 1155 raft_consensus.cc:357] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.690500 1155 raft_consensus.cc:383] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.690546 1155 raft_consensus.cc:738] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.690707 1155 consensus_queue.cc:260] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.691376 877 tablet_service.cc:1468] Processing CreateTablet for tablet 0cf193900a2842e4983a00c8a4f301a9 (DEFAULT_TABLE table=default.Table_5b6cb001443c433a8a3b15de2a24fd9d [id=75722b4530384e63a406aee1edb7b237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.691960 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf193900a2842e4983a00c8a4f301a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.695268 1155 ts_tablet_manager.cc:1428] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:51.695441 1154 tablet_bootstrap.cc:492] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.695518 1155 tablet_bootstrap.cc:492] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.695544 1154 ts_tablet_manager.cc:1397] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.002s
I20250629 01:51:51.696053 1154 raft_consensus.cc:357] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.696220 1154 raft_consensus.cc:383] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.696269 1154 raft_consensus.cc:738] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.696451 1154 consensus_queue.cc:260] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.696884 1155 tablet_bootstrap.cc:654] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.696875 1154 ts_tablet_manager.cc:1428] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:51.697109 1154 tablet_bootstrap.cc:492] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.698421 1154 tablet_bootstrap.cc:654] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.702543 881 tablet_service.cc:1468] Processing CreateTablet for tablet dee14f26866c48cfbf66ee83c8d7ccd4 (DEFAULT_TABLE table=default.Table_09dc040be7aa44f18e37086a158aad84 [id=8e6efb0283f049399851b5bb192431b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.702813 1157 tablet_replica.cc:331] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.703533 1157 raft_consensus.cc:2241] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.703558 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dee14f26866c48cfbf66ee83c8d7ccd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.703666 1157 raft_consensus.cc:2270] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.703984 1159 tablet_bootstrap.cc:492] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.704118 1159 ts_tablet_manager.cc:1397] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250629 01:51:51.704319 878 tablet_service.cc:1468] Processing CreateTablet for tablet b5876573ad1e40c4be3c8fbe61dc9bd5 (DEFAULT_TABLE table=default.Table_eb8ae50799eb4f4693b4df01e18e82d5 [id=6d493092423c43619fd95fb227452cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.704624 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5876573ad1e40c4be3c8fbe61dc9bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.704674 1159 raft_consensus.cc:357] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.704818 1159 raft_consensus.cc:383] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.704872 1159 raft_consensus.cc:738] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.705067 1159 consensus_queue.cc:260] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.705446 1159 ts_tablet_manager.cc:1428] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:51.706306 1155 tablet_bootstrap.cc:492] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.706416 1155 ts_tablet_manager.cc:1397] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20250629 01:51:51.704754 1157 ts_tablet_manager.cc:1905] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.710430 1155 raft_consensus.cc:357] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.710572 1155 raft_consensus.cc:383] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.710633 1155 raft_consensus.cc:738] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.710793 1155 consensus_queue.cc:260] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.711282 1155 ts_tablet_manager.cc:1428] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:51.711570 1155 tablet_bootstrap.cc:492] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.712345 1159 tablet_bootstrap.cc:492] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.713449 1159 tablet_bootstrap.cc:654] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.714212 1026 tablet_service.cc:1468] Processing CreateTablet for tablet df0fe7f5469d44fc8f6ac5c72377b48a (DEFAULT_TABLE table=default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.714540 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0fe7f5469d44fc8f6ac5c72377b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.715543 1158 ts_tablet_manager.cc:1918] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.715649 1158 log.cc:1199] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/35990d206ba64159ab05ab7b7b19d064
I20250629 01:51:51.715885 1158 ts_tablet_manager.cc:1939] T 35990d206ba64159ab05ab7b7b19d064 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.716753 1158 tablet_replica.cc:331] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.716931 1158 raft_consensus.cc:2241] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.717069 1158 raft_consensus.cc:2270] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.717892 1158 ts_tablet_manager.cc:1905] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.718246 1155 tablet_bootstrap.cc:654] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.724714 1157 ts_tablet_manager.cc:1918] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.724803 1157 log.cc:1199] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d91fa0f049904a1fb5cbd38d1fd1693a
I20250629 01:51:51.725834 1158 ts_tablet_manager.cc:1918] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.725930 1158 log.cc:1199] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/432680cae61946af9859760fe23110ec
I20250629 01:51:51.726114 1155 tablet_bootstrap.cc:492] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.726171 1158 ts_tablet_manager.cc:1939] T 432680cae61946af9859760fe23110ec P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.726220 1155 ts_tablet_manager.cc:1397] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20250629 01:51:51.726827 1158 tablet_replica.cc:331] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.726964 1158 raft_consensus.cc:2241] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.727075 1158 raft_consensus.cc:2270] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.727332 1155 raft_consensus.cc:357] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.727468 1155 raft_consensus.cc:383] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.727509 1155 raft_consensus.cc:738] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.727651 1155 consensus_queue.cc:260] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.728013 1155 ts_tablet_manager.cc:1428] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:51.729496 1154 tablet_bootstrap.cc:492] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.729637 1154 ts_tablet_manager.cc:1397] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250629 01:51:51.730193 1158 ts_tablet_manager.cc:1905] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.730240 1154 raft_consensus.cc:357] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.730394 1154 raft_consensus.cc:383] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.730451 1154 raft_consensus.cc:738] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.730672 1154 consensus_queue.cc:260] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.731130 1154 ts_tablet_manager.cc:1428] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:51.732214 1157 ts_tablet_manager.cc:1939] T d91fa0f049904a1fb5cbd38d1fd1693a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.732520 1154 tablet_bootstrap.cc:492] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.733956 1154 tablet_bootstrap.cc:654] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.736471 1155 tablet_bootstrap.cc:492] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.738127 1158 ts_tablet_manager.cc:1918] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.739559 1154 tablet_bootstrap.cc:492] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.739648 1154 ts_tablet_manager.cc:1397] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:51.740998 1155 tablet_bootstrap.cc:654] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.742652 1154 raft_consensus.cc:357] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.742779 1154 raft_consensus.cc:383] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.742830 1154 raft_consensus.cc:738] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.743319 1154 consensus_queue.cc:260] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.743693 1154 ts_tablet_manager.cc:1428] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:51:51.744963 1155 tablet_bootstrap.cc:492] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.745069 1155 ts_tablet_manager.cc:1397] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250629 01:51:51.745602 1155 raft_consensus.cc:357] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.745728 1155 raft_consensus.cc:383] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.745770 1155 raft_consensus.cc:738] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.745913 1155 consensus_queue.cc:260] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.746059 1158 log.cc:1199] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/241a5c9aee3f4297b67b18380ef5ef88
I20250629 01:51:51.746321 1158 ts_tablet_manager.cc:1939] T 241a5c9aee3f4297b67b18380ef5ef88 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.747362 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d91fa0f049904a1fb5cbd38d1fd1693a (table default.Table_1fb69ddfd0b2426abea3a0352dddb3a7 [id=50b11e4f7e8c49de8a781b273a593358]) successfully deleted
W20250629 01:51:51.747989 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 40266441c3684726a58a5029779395fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40266441c3684726a58a5029779395fc already in progress: creating tablet
W20250629 01:51:51.748526 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 543242e4a5064268ae1e5519b569f682 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 543242e4a5064268ae1e5519b569f682 already in progress: creating tablet
W20250629 01:51:51.749243 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5ab0aa4e74d345b2a214b90b5018c82b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ab0aa4e74d345b2a214b90b5018c82b already in progress: creating tablet
I20250629 01:51:51.749608 750 tablet_service.cc:1468] Processing CreateTablet for tablet a0002dca4db24405afdce3e1076eb52e (DEFAULT_TABLE table=default.Table_46cfbca358ce43a99a1a1dc43600e2b1 [id=91daa3f10462457ea1270f4f7bf7257d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.750046 1154 tablet_bootstrap.cc:492] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.750020 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0002dca4db24405afdce3e1076eb52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.751291 1154 tablet_bootstrap.cc:654] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:51.753118 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 845f2422d6494c95a4738f3062c216c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 845f2422d6494c95a4738f3062c216c7 already in progress: creating tablet
I20250629 01:51:51.753186 1155 ts_tablet_manager.cc:1428] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250629 01:51:51.753702 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5c1d273c70ac4db7a215f02ecac11154 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c1d273c70ac4db7a215f02ecac11154 already in progress: creating tablet
I20250629 01:51:51.753890 1155 tablet_bootstrap.cc:492] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:51.754230 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7eb02d3ff1104f6885ea1736852bbe57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eb02d3ff1104f6885ea1736852bbe57 already in progress: creating tablet
W20250629 01:51:51.754758 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c35ba7564d5a4e53bbe91f2b50ac8ad5 already in progress: creating tablet
I20250629 01:51:51.755170 1155 tablet_bootstrap.cc:654] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:51.755249 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9bbbeb43473a4f6f892056a30f62589f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bbbeb43473a4f6f892056a30f62589f already in progress: creating tablet
I20250629 01:51:51.755532 1157 tablet_replica.cc:331] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.755733 1157 raft_consensus.cc:2241] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.755949 1157 raft_consensus.cc:2270] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.756582 1157 ts_tablet_manager.cc:1905] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.757352 1158 tablet_replica.cc:331] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.757504 1158 raft_consensus.cc:2241] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.757625 1158 raft_consensus.cc:2270] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.758244 1029 tablet_service.cc:1468] Processing CreateTablet for tablet a0002dca4db24405afdce3e1076eb52e (DEFAULT_TABLE table=default.Table_46cfbca358ce43a99a1a1dc43600e2b1 [id=91daa3f10462457ea1270f4f7bf7257d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.758548 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0002dca4db24405afdce3e1076eb52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.758677 1154 tablet_bootstrap.cc:492] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.758760 1154 ts_tablet_manager.cc:1397] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20250629 01:51:51.759246 1154 raft_consensus.cc:357] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.759354 1154 raft_consensus.cc:383] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.759390 1154 raft_consensus.cc:738] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.760888 1155 tablet_bootstrap.cc:492] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.760979 1155 ts_tablet_manager.cc:1397] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:51.761525 1155 raft_consensus.cc:357] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.761653 1155 raft_consensus.cc:383] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.761698 1155 raft_consensus.cc:738] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.762229 1154 consensus_queue.cc:260] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.762603 1154 ts_tablet_manager.cc:1428] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:51.762848 1154 tablet_bootstrap.cc:492] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.763844 1158 ts_tablet_manager.cc:1905] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.764415 1154 tablet_bootstrap.cc:654] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.765918 1155 consensus_queue.cc:260] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.766763 1155 ts_tablet_manager.cc:1428] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:51.767095 1157 ts_tablet_manager.cc:1918] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.767191 1157 log.cc:1199] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/dd709b557b334434a6c2c4a60a996342
I20250629 01:51:51.767446 1155 tablet_bootstrap.cc:492] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.767454 1157 ts_tablet_manager.cc:1939] T dd709b557b334434a6c2c4a60a996342 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.767900 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet dd709b557b334434a6c2c4a60a996342 (table default.Table_b5f98a84fe7341ac8e6b11b04066323c [id=dd8b9804d0144a879c539614acde7240]) successfully deleted
I20250629 01:51:51.769322 1158 ts_tablet_manager.cc:1918] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.769388 1155 tablet_bootstrap.cc:654] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.769500 1158 log.cc:1199] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e962a020664b42ea918e5952dc43aade
I20250629 01:51:51.769773 1158 ts_tablet_manager.cc:1939] T e962a020664b42ea918e5952dc43aade P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.770628 1158 tablet_replica.cc:331] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.770920 1158 raft_consensus.cc:2241] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.771232 1158 raft_consensus.cc:2270] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.771798 1158 ts_tablet_manager.cc:1905] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.774811 1915 raft_consensus.cc:491] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.774933 1915 raft_consensus.cc:513] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.775504 1915 leader_election.cc:290] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.776369 1154 tablet_bootstrap.cc:492] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:51:51.776378 850 leader_election.cc:343] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0a574f95ac2a4193a563e63cbe120768
I20250629 01:51:51.776475 1154 ts_tablet_manager.cc:1397] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
W20250629 01:51:51.776973 850 leader_election.cc:343] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0a574f95ac2a4193a563e63cbe120768
I20250629 01:51:51.777091 850 leader_election.cc:304] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.777494 1915 raft_consensus.cc:2747] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.777668 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 0467b153057f491e81681fa399a2a1a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.777832 1154 raft_consensus.cc:357] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.777980 1154 raft_consensus.cc:383] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.778043 1154 raft_consensus.cc:738] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.778070 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0cf193900a2842e4983a00c8a4f301a9 (DEFAULT_TABLE table=default.Table_5b6cb001443c433a8a3b15de2a24fd9d [id=75722b4530384e63a406aee1edb7b237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.778389 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf193900a2842e4983a00c8a4f301a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.779438 1155 tablet_bootstrap.cc:492] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.779528 1155 ts_tablet_manager.cc:1397] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250629 01:51:51.780081 1155 raft_consensus.cc:357] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.780241 1155 raft_consensus.cc:383] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.780277 1158 ts_tablet_manager.cc:1918] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:51.780285 1155 raft_consensus.cc:738] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.780364 1158 log.cc:1199] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5b4701123dfa467ebb6c95f9742b3874
I20250629 01:51:51.780452 1155 consensus_queue.cc:260] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.780650 1158 ts_tablet_manager.cc:1939] T 5b4701123dfa467ebb6c95f9742b3874 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.780889 1155 ts_tablet_manager.cc:1428] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20250629 01:51:51.781018 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2540a84a922949e3adb90d0cded1c951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2540a84a922949e3adb90d0cded1c951 already in progress: creating tablet
W20250629 01:51:51.781769 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7fdabc198d4b48ebb82cd8a889f7dedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fdabc198d4b48ebb82cd8a889f7dedf already in progress: creating tablet
I20250629 01:51:51.781953 1154 consensus_queue.cc:260] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.782174 1157 tablet_replica.cc:331] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.782302 1157 raft_consensus.cc:2241] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:51.782368 1154 ts_tablet_manager.cc:1428] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250629 01:51:51.782543 1157 raft_consensus.cc:2270] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.782574 1154 tablet_bootstrap.cc:492] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.783825 1157 ts_tablet_manager.cc:1905] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.784633 1154 tablet_bootstrap.cc:654] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.785811 1158 tablet_replica.cc:331] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.785948 1158 raft_consensus.cc:2241] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.786056 1158 raft_consensus.cc:2270] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.786465 1923 raft_consensus.cc:491] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.786600 1923 raft_consensus.cc:513] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.786880 1155 tablet_bootstrap.cc:492] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.787060 1923 leader_election.cc:290] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.787114 1158 ts_tablet_manager.cc:1905] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.787575 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47d4287317ec4788bd96ad174c31748d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.787950 988 leader_election.cc:343] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.788163 1155 tablet_bootstrap.cc:654] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.788334 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47d4287317ec4788bd96ad174c31748d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:51.788697 986 leader_election.cc:343] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.788774 986 leader_election.cc:304] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.789310 1923 raft_consensus.cc:2747] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.794324 1155 tablet_bootstrap.cc:492] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.794450 1155 ts_tablet_manager.cc:1397] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20250629 01:51:51.795162 1155 raft_consensus.cc:357] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.795339 1155 raft_consensus.cc:383] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.795403 1155 raft_consensus.cc:738] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.795574 1155 consensus_queue.cc:260] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.795647 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b5876573ad1e40c4be3c8fbe61dc9bd5 (DEFAULT_TABLE table=default.Table_eb8ae50799eb4f4693b4df01e18e82d5 [id=6d493092423c43619fd95fb227452cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.795929 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5876573ad1e40c4be3c8fbe61dc9bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.796511 1154 tablet_bootstrap.cc:492] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.796608 1154 ts_tablet_manager.cc:1397] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20250629 01:51:51.796641 743 tablet_service.cc:1468] Processing CreateTablet for tablet dee14f26866c48cfbf66ee83c8d7ccd4 (DEFAULT_TABLE table=default.Table_09dc040be7aa44f18e37086a158aad84 [id=8e6efb0283f049399851b5bb192431b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.796897 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dee14f26866c48cfbf66ee83c8d7ccd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.797230 1157 ts_tablet_manager.cc:1918] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:51.797303 1157 log.cc:1199] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5b4701123dfa467ebb6c95f9742b3874
I20250629 01:51:51.797583 1157 ts_tablet_manager.cc:1939] T 5b4701123dfa467ebb6c95f9742b3874 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.797746 1154 raft_consensus.cc:357] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.797863 1154 raft_consensus.cc:383] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.797909 1154 raft_consensus.cc:738] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.798003 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
I20250629 01:51:51.798053 1154 consensus_queue.cc:260] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.798527 1154 ts_tablet_manager.cc:1428] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:51.800158 1155 ts_tablet_manager.cc:1428] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250629 01:51:51.800516 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 557ecee0d6e84f77ad10bbdd27859ee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 557ecee0d6e84f77ad10bbdd27859ee3 already in progress: creating tablet
I20250629 01:51:51.800769 1155 tablet_bootstrap.cc:492] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.800716 1030 tablet_service.cc:1468] Processing CreateTablet for tablet dee14f26866c48cfbf66ee83c8d7ccd4 (DEFAULT_TABLE table=default.Table_09dc040be7aa44f18e37086a158aad84 [id=8e6efb0283f049399851b5bb192431b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.800910 1157 tablet_replica.cc:331] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.801033 1157 raft_consensus.cc:2241] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.801151 1157 raft_consensus.cc:2270] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.802318 1157 ts_tablet_manager.cc:1905] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.801811 1154 tablet_bootstrap.cc:492] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.802950 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dee14f26866c48cfbf66ee83c8d7ccd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.803218 738 tablet_service.cc:1468] Processing CreateTablet for tablet b5876573ad1e40c4be3c8fbe61dc9bd5 (DEFAULT_TABLE table=default.Table_eb8ae50799eb4f4693b4df01e18e82d5 [id=6d493092423c43619fd95fb227452cc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.803510 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5876573ad1e40c4be3c8fbe61dc9bd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:51.804036 1158 ts_tablet_manager.cc:1918] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.804662 1158 log.cc:1199] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/75ce1bd4af434f7e99092aafc0ab0118
I20250629 01:51:51.805032 1158 ts_tablet_manager.cc:1939] T 75ce1bd4af434f7e99092aafc0ab0118 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.805294 1154 tablet_bootstrap.cc:654] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.811306 1923 raft_consensus.cc:491] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.811400 1923 raft_consensus.cc:513] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.811856 1923 leader_election.cc:290] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:51.812538 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942f7bbe29f94083b2494aab8fbcc0c5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.813022 1154 tablet_bootstrap.cc:492] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:51:51.812983 988 leader_election.cc:343] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.813133 1154 ts_tablet_manager.cc:1397] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250629 01:51:51.813720 1154 raft_consensus.cc:357] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.813850 1154 raft_consensus.cc:383] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.813905 1154 raft_consensus.cc:738] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.814061 1154 consensus_queue.cc:260] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.816040 1154 ts_tablet_manager.cc:1428] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:51.816272 1154 tablet_bootstrap.cc:492] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.817945 1850 raft_consensus.cc:491] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.818327 1154 tablet_bootstrap.cc:654] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.819021 1157 ts_tablet_manager.cc:1918] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.819484 1157 log.cc:1199] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/69980d5bf37d4001a6196b22c34ccc94
I20250629 01:51:51.819901 1157 ts_tablet_manager.cc:1939] T 69980d5bf37d4001a6196b22c34ccc94 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.820749 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 69980d5bf37d4001a6196b22c34ccc94 (table default.Table_ae2d1a922b914ba1bfa2dbca0cda5013 [id=fd2351d583834130bb8213424bf96abb]) successfully deleted
W20250629 01:51:51.821681 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3c70f54de70b4d0e94d25239b097f8fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c70f54de70b4d0e94d25239b097f8fa already in progress: creating tablet
I20250629 01:51:51.822254 1157 tablet_replica.cc:331] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.822683 1157 raft_consensus.cc:2241] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.823210 1157 raft_consensus.cc:2270] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.824355 1157 ts_tablet_manager.cc:1905] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.824630 1154 tablet_bootstrap.cc:492] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.824729 1154 ts_tablet_manager.cc:1397] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20250629 01:51:51.825255 1154 raft_consensus.cc:357] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.825381 1154 raft_consensus.cc:383] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.825426 1154 raft_consensus.cc:738] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.826650 1154 consensus_queue.cc:260] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.827742 1154 ts_tablet_manager.cc:1428] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:51:51.828263 1154 tablet_bootstrap.cc:492] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.830511 1154 tablet_bootstrap.cc:654] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.833106 1159 tablet_bootstrap.cc:492] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:51.833300 1159 ts_tablet_manager.cc:1397] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.121s user 0.020s sys 0.009s
I20250629 01:51:51.834340 1159 raft_consensus.cc:357] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.834597 1159 raft_consensus.cc:383] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.834698 1159 raft_consensus.cc:738] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.834954 1159 consensus_queue.cc:260] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:51.835510 1159 ts_tablet_manager.cc:1428] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:51.835937 1154 tablet_bootstrap.cc:492] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.836040 1154 ts_tablet_manager.cc:1397] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250629 01:51:51.836252 1159 tablet_bootstrap.cc:492] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:51.836593 1154 raft_consensus.cc:357] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.836716 1154 raft_consensus.cc:383] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.836757 1154 raft_consensus.cc:738] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.836897 1154 consensus_queue.cc:260] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.837229 1154 ts_tablet_manager.cc:1428] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:51.837637 1159 tablet_bootstrap.cc:654] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.839370 1154 tablet_bootstrap.cc:492] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.840718 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 35990d206ba64159ab05ab7b7b19d064 (table default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 [id=8d49193b68504e77818e373ecad346b6]) successfully deleted
I20250629 01:51:51.840970 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 432680cae61946af9859760fe23110ec (table default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 [id=2dfa777b5eab4acb9adfd35ab56c17d0]) successfully deleted
I20250629 01:51:51.841131 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 241a5c9aee3f4297b67b18380ef5ef88 (table default.Table_9b82f19de1c74fcd9973b4a4084712e1 [id=a9617884dddf40ee883c482c170d8a22]) successfully deleted
I20250629 01:51:51.841287 1154 tablet_bootstrap.cc:654] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:51.841316 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2540a84a922949e3adb90d0cded1c951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2540a84a922949e3adb90d0cded1c951 already in progress: creating tablet
I20250629 01:51:51.841569 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e962a020664b42ea918e5952dc43aade (table default.Table_e0533c0c40ca4af1be6ac30ee37f3aed [id=1d5fd1e402a54d31b3fea01d38db6918]) successfully deleted
I20250629 01:51:51.841737 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5b4701123dfa467ebb6c95f9742b3874 (table default.Table_fb9fc3beabf04d8d94b0b64c4e12139d [id=650823b1aa1448e69087a7de58f9532e]) successfully deleted
W20250629 01:51:51.841950 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7fdabc198d4b48ebb82cd8a889f7dedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fdabc198d4b48ebb82cd8a889f7dedf already in progress: creating tablet
W20250629 01:51:51.842156 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 557ecee0d6e84f77ad10bbdd27859ee3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 557ecee0d6e84f77ad10bbdd27859ee3 already in progress: creating tablet
W20250629 01:51:51.842344 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3c70f54de70b4d0e94d25239b097f8fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c70f54de70b4d0e94d25239b097f8fa already in progress: creating tablet
I20250629 01:51:51.842618 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 75ce1bd4af434f7e99092aafc0ab0118 (table default.Table_d9bb8cc1cab544148faa53db857172d3 [id=5d8dd5f1acfd4c57b7080fc663a1d32d]) successfully deleted
W20250629 01:51:51.842813 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 352a8dbd7f164ca9b622fed30bc6beeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352a8dbd7f164ca9b622fed30bc6beeb already in progress: creating tablet
W20250629 01:51:51.843007 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7908afa2abaa4521b44f954e0790f986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7908afa2abaa4521b44f954e0790f986 already in progress: creating tablet
W20250629 01:51:51.843211 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0467b153057f491e81681fa399a2a1a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0467b153057f491e81681fa399a2a1a9 already in progress: creating tablet
I20250629 01:51:51.844285 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a20245e23a9e4dff9bfb1e565dc38184
I20250629 01:51:51.844642 449 catalog_manager.cc:6652] ReplaceTablet: tablet a20245e23a9e4dff9bfb1e565dc38184 deleted and replaced by tablet 29e05dd9908c4390a55b55e2eef8d4b2
I20250629 01:51:51.847491 1154 tablet_bootstrap.cc:492] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.847576 1154 ts_tablet_manager.cc:1397] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250629 01:51:51.848999 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_abe281abbab94551a471e46d3908fc39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.849179 1154 raft_consensus.cc:357] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.849289 1154 raft_consensus.cc:383] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.849325 1154 raft_consensus.cc:738] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:51.849589 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abe281abbab94551a471e46d3908fc39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.849882 1154 consensus_queue.cc:260] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.850255 1154 ts_tablet_manager.cc:1428] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:51.850461 1154 tablet_bootstrap.cc:492] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.850906 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 57f7f71ab5e54d249141c89d9e806111
I20250629 01:51:51.851519 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_985b5a9e7b454754a12e4e3fdb38a3a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.852041 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_985b5a9e7b454754a12e4e3fdb38a3a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:51.853363 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 consensus state has no leader
I20250629 01:51:51.853721 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5beabed0e3e44bfe825659e35faa85fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.854197 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5beabed0e3e44bfe825659e35faa85fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.854605 1154 tablet_bootstrap.cc:654] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.855425 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8975c1deaaf44761be791f460b53ec74
I20250629 01:51:51.856065 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_614ed7de687c4876868bd54af35e3f8a" } new_table_name: "default.Table_d9c2bb1acc984b17a6f7ea9ed229be68" modify_external_catalogs: true
W20250629 01:51:51.857335 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a873bc03533a405185eca6ad4bb5db3a consensus state has no leader
I20250629 01:51:51.857565 451 master_service.cc:948] ReplaceTablet: received request to replace tablet faca0e904a04456d9aead23abd44e00a from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:51.858808 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bae91d28f27f4e14a23c65e41091b671 consensus state has no leader
I20250629 01:51:51.859159 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0cf193900a2842e4983a00c8a4f301a9 (DEFAULT_TABLE table=default.Table_5b6cb001443c433a8a3b15de2a24fd9d [id=75722b4530384e63a406aee1edb7b237]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:51.859145 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3a542c87991a41379b067ca98b5235fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:51.859500 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf193900a2842e4983a00c8a4f301a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:51.859618 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a542c87991a41379b067ca98b5235fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.818074 1850 raft_consensus.cc:513] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.861384 1154 tablet_bootstrap.cc:492] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.861474 1154 ts_tablet_manager.cc:1397] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250629 01:51:51.861985 1154 raft_consensus.cc:357] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.863098 1154 raft_consensus.cc:383] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.863147 1154 raft_consensus.cc:738] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.863288 1154 consensus_queue.cc:260] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.863636 1154 ts_tablet_manager.cc:1428] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:51.863818 1154 tablet_bootstrap.cc:492] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.865486 1154 tablet_bootstrap.cc:654] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.868827 1824 raft_consensus.cc:491] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.870536 745 tablet_service.cc:1515] Processing DeleteTablet for tablet a20245e23a9e4dff9bfb1e565dc38184 with delete_type TABLET_DATA_DELETED (replaced by tablet 29e05dd9908c4390a55b55e2eef8d4b2) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.870921 1154 tablet_bootstrap.cc:492] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:51.871002 1154 ts_tablet_manager.cc:1397] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250629 01:51:51.871531 1154 raft_consensus.cc:357] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.871636 1154 raft_consensus.cc:383] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.871670 1154 raft_consensus.cc:738] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.871817 1154 consensus_queue.cc:260] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.872237 1154 ts_tablet_manager.cc:1428] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:51.872440 1154 tablet_bootstrap.cc:492] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:51.873564 1154 tablet_bootstrap.cc:654] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.877226 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.877810 712 leader_election.cc:343] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.882062 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_12a83e36ffff41fa944eefd75cdc2bf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.883414 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12a83e36ffff41fa944eefd75cdc2bf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.891935 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a20245e23a9e4dff9bfb1e565dc38184 with delete_type TABLET_DATA_DELETED (replaced by tablet 29e05dd9908c4390a55b55e2eef8d4b2) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.892643 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b16b6840c24484b90218fffc1da9621 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.893447 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f7f71ab5e54d249141c89d9e806111 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.893709 1158 tablet_replica.cc:331] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.893865 1158 raft_consensus.cc:2241] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.894006 1158 raft_consensus.cc:2270] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.893999 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:51.894204 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0467b153057f491e81681fa399a2a1a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.894435 1158 ts_tablet_manager.cc:1905] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.895495 1156 tablet_replica.cc:331] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.896275 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8975c1deaaf44761be791f460b53ec74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.896971 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 47d4287317ec4788bd96ad174c31748d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.897439 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8799b1858bf2408a85f0956578639393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.900187 1158 ts_tablet_manager.cc:1918] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250629 01:51:51.876394 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942f7bbe29f94083b2494aab8fbcc0c5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:51.900290 1158 log.cc:1199] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/57f7f71ab5e54d249141c89d9e806111
I20250629 01:51:51.900599 1158 ts_tablet_manager.cc:1939] T 57f7f71ab5e54d249141c89d9e806111 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:51.900660 986 leader_election.cc:343] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.900883 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 8975c1deaaf44761be791f460b53ec74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.900827 986 leader_election.cc:304] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:51.901252 1850 leader_election.cc:290] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.901260 1923 raft_consensus.cc:2747] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.901401 1850 raft_consensus.cc:491] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.901515 1158 tablet_replica.cc:331] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.901485 1850 raft_consensus.cc:513] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.901674 1158 raft_consensus.cc:2241] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.901494 745 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5150eff1ad43588a9e70dab47744e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bef2747f0c04f498079906e40bac24c) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.901783 1158 raft_consensus.cc:2270] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.900182 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24d0e668c49ac825b2578849ab2a3) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.902453 1158 ts_tablet_manager.cc:1905] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.902894 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.903461 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.904175 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.904738 712 leader_election.cc:343] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.906850 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.907364 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.907852 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.910209 1158 ts_tablet_manager.cc:1918] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.910306 1158 log.cc:1199] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/47d4287317ec4788bd96ad174c31748d
I20250629 01:51:51.910566 1158 ts_tablet_manager.cc:1939] T 47d4287317ec4788bd96ad174c31748d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.911317 1158 tablet_replica.cc:331] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.911479 1158 raft_consensus.cc:2241] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.911610 1158 raft_consensus.cc:2270] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.913789 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.914415 1158 ts_tablet_manager.cc:1905] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.917711 1850 leader_election.cc:290] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.918257 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f7f71ab5e54d249141c89d9e806111 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.902033 1824 raft_consensus.cc:513] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.918852 1156 raft_consensus.cc:2241] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.920604 1824 leader_election.cc:290] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.921279 1156 raft_consensus.cc:2270] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.921844 1156 ts_tablet_manager.cc:1905] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.923023 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0b16b6840c24484b90218fffc1da9621" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:51.923504 1155 tablet_bootstrap.cc:654] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:51.923534 712 leader_election.cc:343] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.898901 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6708008981374868897e9b1e1d6dba18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.928017 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6708008981374868897e9b1e1d6dba18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.920274 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.899127 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_324620c729034dc2bd24e044a719599f" } modify_external_catalogs: true
I20250629 01:51:51.929328 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_324620c729034dc2bd24e044a719599f" } modify_external_catalogs: true
I20250629 01:51:51.930384 1156 ts_tablet_manager.cc:1918] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.930544 1156 log.cc:1199] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0b16b6840c24484b90218fffc1da9621
I20250629 01:51:51.930893 1156 ts_tablet_manager.cc:1939] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:51.932410 1156 tablet_replica.cc:331] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:51.932861 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2e7ef5453b1945728384798f25e47b7f" } modify_external_catalogs: true
I20250629 01:51:51.933087 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2e7ef5453b1945728384798f25e47b7f" } modify_external_catalogs: true
I20250629 01:51:51.933303 1156 raft_consensus.cc:2241] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.933516 1156 raft_consensus.cc:2270] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.934091 1156 ts_tablet_manager.cc:1905] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:51.935891 1923 raft_consensus.cc:491] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:51.936028 1923 raft_consensus.cc:513] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.936662 1923 leader_election.cc:290] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:51.937281 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd084685cc0407da9e12b0a4df386e9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:51.937793 986 leader_election.cc:343] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.938087 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd084685cc0407da9e12b0a4df386e9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:51.938645 988 leader_election.cc:343] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.938752 988 leader_election.cc:304] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:51.938979 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet bae91d28f27f4e14a23c65e41091b671 (table default.Table_884f76ae76564136b99617ba1c1ca0a8 [id=c968db78da2c44a6900dc90d8c395c2d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bae91d28f27f4e14a23c65e41091b671 consensus state has no leader
I20250629 01:51:51.939182 1923 raft_consensus.cc:2747] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:51.939875 1158 ts_tablet_manager.cc:1918] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.939958 1158 log.cc:1199] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8799b1858bf2408a85f0956578639393
I20250629 01:51:51.940196 1158 ts_tablet_manager.cc:1939] T 8799b1858bf2408a85f0956578639393 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.941330 1157 ts_tablet_manager.cc:1918] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.941455 1157 log.cc:1199] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0a574f95ac2a4193a563e63cbe120768
I20250629 01:51:51.941721 1157 ts_tablet_manager.cc:1939] T 0a574f95ac2a4193a563e63cbe120768 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:51.942163 1157 tablet_replica.cc:331] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:51.942466 1157 raft_consensus.cc:2241] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.942613 1157 raft_consensus.cc:2270] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.943670 1157 ts_tablet_manager.cc:1905] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:51.943856 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 48522.4us
W20250629 01:51:51.944304 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a20245e23a9e4dff9bfb1e565dc38184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20245e23a9e4dff9bfb1e565dc38184 already in progress: creating tablet
I20250629 01:51:51.944787 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_ab9e746444a24266aca5439c51b30a27 [id=60094a0a7c134476a2a4e44d8555c5d6]) successfully deleted
W20250629 01:51:51.945106 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8975c1deaaf44761be791f460b53ec74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8975c1deaaf44761be791f460b53ec74 already in progress: creating tablet
I20250629 01:51:51.945358 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 47d4287317ec4788bd96ad174c31748d (table default.Table_58a60b60dab64affa9d8b5b1acf53326 [id=d524cd9f7aba4981b939a770099325c8]) successfully deleted
I20250629 01:51:51.945417 1158 tablet_replica.cc:331] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.945611 1158 raft_consensus.cc:2241] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.945654 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8799b1858bf2408a85f0956578639393 (table default.Table_1b80dba0f8d04043b95a7aaa176636b9 [id=511866fda7ae4bde9cea9777b36cd76f]) successfully deleted
I20250629 01:51:51.945736 1158 raft_consensus.cc:2270] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.947475 1158 ts_tablet_manager.cc:1905] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:51.948722 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 77864us
I20250629 01:51:51.949470 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:51.949637 1049 raft_consensus.cc:2466] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:51.951522 710 leader_election.cc:304] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:51.952215 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:51.952366 1048 raft_consensus.cc:2466] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:51.952507 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 with delete_type TABLET_DATA_DELETED (replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.952943 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 34172d05ae40417494cd8f67f4cb981e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.953269 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:51.953586 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a20245e23a9e4dff9bfb1e565dc38184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20245e23a9e4dff9bfb1e565dc38184 already in progress: creating tablet
I20250629 01:51:51.953981 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0b16b6840c24484b90218fffc1da9621 (table default.Table_10a0ae1fcff049dda2c11881c3191fa3 [id=14974743edd741fbab51f9e8a4add096]) successfully deleted
W20250629 01:51:51.954164 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0467b153057f491e81681fa399a2a1a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0467b153057f491e81681fa399a2a1a9 already in progress: creating tablet
W20250629 01:51:51.954322 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8975c1deaaf44761be791f460b53ec74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8975c1deaaf44761be791f460b53ec74 already in progress: creating tablet
I20250629 01:51:51.954625 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a20245e23a9e4dff9bfb1e565dc38184 with delete_type TABLET_DATA_DELETED (replaced by tablet 29e05dd9908c4390a55b55e2eef8d4b2) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.955080 881 tablet_service.cc:1515] Processing DeleteTablet for tablet be8e0946a20340b38835c4d8daeaa7b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.955776 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f7f71ab5e54d249141c89d9e806111 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.956535 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9587986b83104f2aabf6ef98ebad9e37" } new_table_name: "default.Table_65704a9f46f54a6d8ac7cd55be389d28" modify_external_catalogs: true
I20250629 01:51:51.958130 455 master_service.cc:948] ReplaceTablet: received request to replace tablet c5e9646c3ad04ed580f1d3402010dd38 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:51.958473 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d87639255ec46d98fcfb0f9638ee39a consensus state has no leader
I20250629 01:51:51.958797 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_aaa30d8b97a8406fb1f5fe4c5f309010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:51.959234 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaa30d8b97a8406fb1f5fe4c5f309010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:51.959403 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 8975c1deaaf44761be791f460b53ec74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.959602 1861 raft_consensus.cc:2802] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:51.959666 1861 raft_consensus.cc:491] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:51.959726 1861 raft_consensus.cc:3058] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.960366 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet faca0e904a04456d9aead23abd44e00a
I20250629 01:51:51.960669 1158 ts_tablet_manager.cc:1918] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.960685 451 catalog_manager.cc:6652] ReplaceTablet: tablet faca0e904a04456d9aead23abd44e00a deleted and replaced by tablet 4de6fca85fb643b998be42e9bc7bdbb6
I20250629 01:51:51.960740 1158 log.cc:1199] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:51.960886 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c208a66223c24a10abbc5478f7469f04" } modify_external_catalogs: true
I20250629 01:51:51.960954 1158 ts_tablet_manager.cc:1939] T 9cf553b291854a6f9e7ed78b5db5f18c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.961009 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c208a66223c24a10abbc5478f7469f04" } modify_external_catalogs: true
I20250629 01:51:51.962117 1861 raft_consensus.cc:513] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.963006 1861 leader_election.cc:290] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.963021 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:51.963387 712 leader_election.cc:343] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.965231 755 tablet_service.cc:1515] Processing DeleteTablet for tablet faca0e904a04456d9aead23abd44e00a with delete_type TABLET_DATA_DELETED (replaced by tablet 4de6fca85fb643b998be42e9bc7bdbb6) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:51.967938 1158 tablet_replica.cc:331] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.968113 1158 raft_consensus.cc:2241] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.968233 1158 raft_consensus.cc:2270] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.969206 1158 ts_tablet_manager.cc:1905] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:51.972409 710 leader_election.cc:343] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0b16b6840c24484b90218fffc1da9621
I20250629 01:51:51.972523 710 leader_election.cc:304] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:51.972707 710 raft_consensus.cc:2684] T 0b16b6840c24484b90218fffc1da9621 P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:51.973708 710 leader_election.cc:304] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:51.973929 1861 raft_consensus.cc:2802] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:51.973995 1861 raft_consensus.cc:491] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:51.974057 1861 raft_consensus.cc:3058] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.976853 1861 raft_consensus.cc:513] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.977746 1861 leader_election.cc:290] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:51.977797 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:51.978081 1158 ts_tablet_manager.cc:1918] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.978174 1158 log.cc:1199] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/897a4cefefe04f06b47070ec7fa1a33b
W20250629 01:51:51.978159 712 leader_election.cc:343] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:51.978426 1158 ts_tablet_manager.cc:1939] T 897a4cefefe04f06b47070ec7fa1a33b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:51.979036 1158 tablet_replica.cc:331] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:51.979180 1158 raft_consensus.cc:2241] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:51.979288 1158 raft_consensus.cc:2270] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:51.979863 1155 tablet_bootstrap.cc:492] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.979993 1155 ts_tablet_manager.cc:1397] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.179s user 0.013s sys 0.013s
I20250629 01:51:51.980808 1155 raft_consensus.cc:357] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.980986 1155 raft_consensus.cc:383] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.981037 1155 raft_consensus.cc:738] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.981221 1155 consensus_queue.cc:260] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.982213 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:51.982283 1155 ts_tablet_manager.cc:1428] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:51.982331 1048 raft_consensus.cc:3058] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.982520 1155 tablet_bootstrap.cc:492] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:51.982517 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:51.982630 1049 raft_consensus.cc:3058] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:51.983791 1155 tablet_bootstrap.cc:654] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:51.984462 1048 raft_consensus.cc:2466] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:51.985016 1049 raft_consensus.cc:2466] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:51.985985 1158 ts_tablet_manager.cc:1905] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:51.987391 712 consensus_peers.cc:489] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:51.987700 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9cf553b291854a6f9e7ed78b5db5f18c (table default.Table_770438a7f7324802ab8dea0697fc2423 [id=d3d515038aa84669a3a64824de88f06e]) successfully deleted
I20250629 01:51:51.987929 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 897a4cefefe04f06b47070ec7fa1a33b (table default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]) successfully deleted
I20250629 01:51:51.988189 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet faca0e904a04456d9aead23abd44e00a with delete_type TABLET_DATA_DELETED (replaced by tablet 4de6fca85fb643b998be42e9bc7bdbb6) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:51.989089 449 master_service.cc:948] ReplaceTablet: received request to replace tablet f3bdc8bda8a940be865679a05b90bd08 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:51.989686 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0a574f95ac2a4193a563e63cbe120768 (table default.Table_399e960d59fe4143843457bd372dbf63 [id=99712a47e4424c78bb4721eeec6e90c3]) successfully deleted
I20250629 01:51:51.990164 881 tablet_service.cc:1515] Processing DeleteTablet for tablet faca0e904a04456d9aead23abd44e00a with delete_type TABLET_DATA_DELETED (replaced by tablet 4de6fca85fb643b998be42e9bc7bdbb6) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:51.990626 1158 ts_tablet_manager.cc:1918] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:51.990731 1158 log.cc:1199] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0d925ffce89b40258b29ec021bfa5f92
I20250629 01:51:51.990984 1158 ts_tablet_manager.cc:1939] T 0d925ffce89b40258b29ec021bfa5f92 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:51.991226 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e16e6adfa2be40a2885267800331f664 consensus state has no leader
I20250629 01:51:51.994745 1155 tablet_bootstrap.cc:492] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:51.994851 1155 ts_tablet_manager.cc:1397] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20250629 01:51:51.995488 1155 raft_consensus.cc:357] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.995644 1155 raft_consensus.cc:383] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:51.995726 1155 raft_consensus.cc:738] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:51.995918 1155 consensus_queue.cc:260] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:51.996405 710 leader_election.cc:304] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:51.999415 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:51.997200 1155 ts_tablet_manager.cc:1428] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:51.999850 1861 raft_consensus.cc:2802] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:51.999964 1861 raft_consensus.cc:695] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:52.000420 1155 tablet_bootstrap.cc:492] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.000231 1861 consensus_queue.cc:237] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.001782 2049 raft_consensus.cc:491] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.001976 2049 raft_consensus.cc:513] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.002661 2049 leader_election.cc:290] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.002682 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0827ec622ff04ca6b2fec50e40b9e1de" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.002801 1155 tablet_bootstrap.cc:654] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:52.003181 712 leader_election.cc:343] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.005155 1158 tablet_replica.cc:331] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.003414 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0d925ffce89b40258b29ec021bfa5f92 (table default.Table_1b37a535cd2f483a8c72aa10882a4301 [id=b516da4148384818bcc69e4259f20ef9]) successfully deleted
I20250629 01:51:52.005321 1158 raft_consensus.cc:2241] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.005466 1158 raft_consensus.cc:2270] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.006454 1158 ts_tablet_manager.cc:1905] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.006510 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bedbb041e4094d8399de625badd971bb
I20250629 01:51:52.006543 710 leader_election.cc:304] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:52.006783 2049 raft_consensus.cc:2802] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:52.006986 2049 raft_consensus.cc:695] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:52.007192 2049 consensus_queue.cc:237] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.008219 755 tablet_service.cc:1515] Processing DeleteTablet for tablet bedbb041e4094d8399de625badd971bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.008994 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet bedbb041e4094d8399de625badd971bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:52.009147 710 leader_election.cc:343] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0827ec622ff04ca6b2fec50e40b9e1de
I20250629 01:51:52.009441 710 leader_election.cc:304] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.009677 2052 raft_consensus.cc:2747] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.011794 1158 ts_tablet_manager.cc:1918] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.011893 1158 log.cc:1199] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/942f7bbe29f94083b2494aab8fbcc0c5
I20250629 01:51:52.012183 1158 ts_tablet_manager.cc:1939] T 942f7bbe29f94083b2494aab8fbcc0c5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.012856 1158 tablet_replica.cc:331] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.012974 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 942f7bbe29f94083b2494aab8fbcc0c5 (table default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]) successfully deleted
I20250629 01:51:52.013013 1158 raft_consensus.cc:2241] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.013139 1158 raft_consensus.cc:2270] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.014735 1156 ts_tablet_manager.cc:1918] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.015766 1154 tablet_bootstrap.cc:492] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.015888 1154 ts_tablet_manager.cc:1397] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.143s user 0.035s sys 0.047s
I20250629 01:51:52.016052 1156 log.cc:1199] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/bd5150eff1ad43588a9e70dab47744e3
I20250629 01:51:52.016530 1154 raft_consensus.cc:357] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.016678 1154 raft_consensus.cc:383] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.016758 1154 raft_consensus.cc:738] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.016958 1154 consensus_queue.cc:260] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.017041 1156 ts_tablet_manager.cc:1939] T bd5150eff1ad43588a9e70dab47744e3 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.017518 1154 ts_tablet_manager.cc:1428] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:52.017800 1154 tablet_bootstrap.cc:492] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.019229 1154 tablet_bootstrap.cc:654] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.020153 1158 ts_tablet_manager.cc:1905] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.020426 1155 tablet_bootstrap.cc:492] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.022760 1155 ts_tablet_manager.cc:1397] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20250629 01:51:52.023303 1155 raft_consensus.cc:357] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.023456 1155 raft_consensus.cc:383] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.023516 1155 raft_consensus.cc:738] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.023737 1155 consensus_queue.cc:260] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.024135 1155 ts_tablet_manager.cc:1428] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:52.024130 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet bd5150eff1ad43588a9e70dab47744e3 (table default.Table_99e79a9aa55444b6b6d1873d0d4844c8 [id=c483797822d647458b5728bf3255b15d]) successfully deleted
I20250629 01:51:52.025072 1155 tablet_bootstrap.cc:492] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.027228 1155 tablet_bootstrap.cc:654] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.028816 1159 tablet_bootstrap.cc:492] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.028923 1159 ts_tablet_manager.cc:1397] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.193s user 0.003s sys 0.004s
I20250629 01:51:52.029507 1158 ts_tablet_manager.cc:1918] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.029608 1158 log.cc:1199] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b0c2dd05f5bb466fbe38ed259ce3a4b8
I20250629 01:51:52.029592 1159 raft_consensus.cc:357] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.029767 1159 raft_consensus.cc:383] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.029829 1159 raft_consensus.cc:738] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.029870 1158 ts_tablet_manager.cc:1939] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:52.013145 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873): Couldn't send request to peer 0f4258c004e04bcf8216bb7c8fd56c54. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250629 01:51:52.031119 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a90b206d58d04acdb1a8cb0c45d68c31 with delete_type TABLET_DATA_DELETED (replaced by tablet bccdcd9de08648bb925db6fe56f97d20) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.031826 1158 tablet_replica.cc:331] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.031977 1158 raft_consensus.cc:2241] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.031994 881 tablet_service.cc:1515] Processing DeleteTablet for tablet bedbb041e4094d8399de625badd971bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.032119 1158 raft_consensus.cc:2270] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.032802 1155 tablet_bootstrap.cc:492] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.032912 1155 ts_tablet_manager.cc:1397] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20250629 01:51:52.033169 1158 ts_tablet_manager.cc:1905] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.033174 2049 consensus_queue.cc:1035] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:52.033527 1155 raft_consensus.cc:357] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.033668 1155 raft_consensus.cc:383] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.033736 1155 raft_consensus.cc:738] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.035089 1155 consensus_queue.cc:260] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.035867 1155 ts_tablet_manager.cc:1428] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:52.036144 1155 tablet_bootstrap.cc:492] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.037752 1159 consensus_queue.cc:260] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.041368 2061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:35873 every 8 connection(s)
I20250629 01:51:52.041844 1158 ts_tablet_manager.cc:1918] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.041966 1158 log.cc:1199] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:52.042263 1158 ts_tablet_manager.cc:1939] T 196f1bd233ab4399a9d22a55b8bc2635 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.043041 1158 tablet_replica.cc:331] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.043251 1158 raft_consensus.cc:2241] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.043407 1158 raft_consensus.cc:2270] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.048063 1851 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:35873
I20250629 01:51:52.048154 1158 ts_tablet_manager.cc:1905] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.050522 1159 ts_tablet_manager.cc:1428] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250629 01:51:52.051093 1851 server_base.cc:1180] Dumped server information to /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
I20250629 01:51:52.055444 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 (table default.Table_d7789b77f4814c388e1eb092297120b1 [id=b890fb8e74dc464bbf195cdfa89814be]) successfully deleted
I20250629 01:51:52.055917 1155 tablet_bootstrap.cc:654] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.056928 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu as pid 1851
I20250629 01:51:52.059193 1158 ts_tablet_manager.cc:1918] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.059299 1158 log.cc:1199] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:52.059561 1158 ts_tablet_manager.cc:1939] T a485e0e6e2f04aa6bd67a62f6d30be26 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.060353 1158 tablet_replica.cc:331] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.060511 1158 raft_consensus.cc:2241] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.060619 1158 raft_consensus.cc:2270] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.063088 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8aceaa540cf94e83ba9250c612cab7a5" } modify_external_catalogs: true
I20250629 01:51:52.063295 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8aceaa540cf94e83ba9250c612cab7a5" } modify_external_catalogs: true
I20250629 01:51:52.064131 449 master_service.cc:948] ReplaceTablet: received request to replace tablet dbfcb9233ace429db5daa593c2a1dcbe from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:52.064189 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_52330030fd4140c1af870dcfcb4dc269" } new_table_name: "default.Table_db86a323f87542f1ac0296e67733fa1d" modify_external_catalogs: true
I20250629 01:51:52.065008 1156 tablet_replica.cc:331] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.065212 1156 raft_consensus.cc:2241] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.065461 1156 raft_consensus.cc:2270] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.066291 1915 raft_consensus.cc:491] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.066321 1156 ts_tablet_manager.cc:1905] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.066421 1915 raft_consensus.cc:513] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.066612 1158 ts_tablet_manager.cc:1905] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.067015 1915 leader_election.cc:290] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.067708 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:52.069214 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 44186.1us
I20250629 01:51:52.072139 1158 ts_tablet_manager.cc:1918] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.072232 1158 log.cc:1199] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:52.072479 1158 ts_tablet_manager.cc:1939] T 34172d05ae40417494cd8f67f4cb981e P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.072876 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3821b9a8d745433382baadfd96b6c260" } modify_external_catalogs: true
I20250629 01:51:52.073028 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3821b9a8d745433382baadfd96b6c260" } modify_external_catalogs: true
I20250629 01:51:52.073150 1158 tablet_replica.cc:331] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.073287 1158 raft_consensus.cc:2241] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.073410 1158 raft_consensus.cc:2270] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.073473 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ac0cef7ef73646d1b010c08c31cff2ff" } new_table_name: "default.Table_86157cacc72642338c2bff2ec0dd2052" modify_external_catalogs: true
I20250629 01:51:52.075487 1155 tablet_bootstrap.cc:492] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.075649 1155 ts_tablet_manager.cc:1397] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20250629 01:51:52.076697 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:52.077286 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_de085cd6f5f84a5eabb5f1d77be715b1" } new_table_name: "default.Table_8b19cb4429164a77b7cd89742caf4afa" modify_external_catalogs: true
I20250629 01:51:52.081153 1158 ts_tablet_manager.cc:1905] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.084825 2065 raft_consensus.cc:491] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.086608 2065 raft_consensus.cc:513] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.087296 2065 leader_election.cc:290] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:52.088757 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce7a72db953646038a9d82dde04d76e4 consensus state has no leader
I20250629 01:51:52.089224 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0d73122281254c95b722afbfdd68d670"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.089787 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d73122281254c95b722afbfdd68d670 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.094377 1159 tablet_bootstrap.cc:492] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.085812 1158 ts_tablet_manager.cc:1918] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.100551 1158 log.cc:1199] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8dd084685cc0407da9e12b0a4df386e9
I20250629 01:51:52.100833 1158 ts_tablet_manager.cc:1939] T 8dd084685cc0407da9e12b0a4df386e9 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.101682 1158 tablet_replica.cc:331] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.101856 1158 raft_consensus.cc:2241] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.102594 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1336aea8f6b1436a971766d92d3c1197
I20250629 01:51:52.103241 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c02b87b2cd724dc9a010ea32cbbb057c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.103677 1861 raft_consensus.cc:491] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:52.103761 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c02b87b2cd724dc9a010ea32cbbb057c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.103812 1861 raft_consensus.cc:513] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.104753 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47d4287317ec4788bd96ad174c31748d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.105156 712 leader_election.cc:343] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.105507 1861 leader_election.cc:290] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.105487 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.105986 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bf612fea5b74a0bb78700cc07c4d68d consensus state has no leader
I20250629 01:51:52.106245 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d4a2315a9cc24966b7cd234220e6c247"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.106635 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4a2315a9cc24966b7cd234220e6c247 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:52.106600 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 70632.7us
I20250629 01:51:52.107820 1915 raft_consensus.cc:491] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.107921 1915 raft_consensus.cc:513] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.108476 1915 leader_election.cc:290] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.108829 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2049e7465da4404a86dbaf5a77adcda6
I20250629 01:51:52.108932 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:52.110914 850 leader_election.cc:343] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 26d85fd3d3dd4b07a812c019e1a5c748
I20250629 01:51:52.111299 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "399f460962134ee0b018e9cb8f5d6d2b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:52.111490 1049 raft_consensus.cc:2391] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 443c4bd3c5d548ee8cd347d2b4de5293 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
W20250629 01:51:52.112613 988 leader_election.cc:343] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.112783 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:52.113006 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:52.113855 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 1336aea8f6b1436a971766d92d3c1197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:52.114068 986 leader_election.cc:343] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.114147 986 leader_election.cc:304] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.114290 1923 raft_consensus.cc:2747] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.114349 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 2049e7465da4404a86dbaf5a77adcda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.116611 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1336aea8f6b1436a971766d92d3c1197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.117213 448 catalog_manager.cc:5582] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:52.117560 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:52.117761 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]). Expected version 4 got 0
I20250629 01:51:52.118026 448 catalog_manager.cc:5582] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:52.118287 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]). Expected version 3 got 0
I20250629 01:51:52.119930 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 2049e7465da4404a86dbaf5a77adcda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.121222 1155 raft_consensus.cc:357] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.121368 1155 raft_consensus.cc:383] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.121414 1155 raft_consensus.cc:738] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.123358 914 raft_consensus.cc:3058] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.123383 881 tablet_service.cc:1468] Processing CreateTablet for tablet 4416bbec7d1c40df9a835b503d368d4c (DEFAULT_TABLE table=default.Table_9396b8200cc74c71b35de340567589a4 [id=09be72ea10df4751b36fda893f0b2406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.123875 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4416bbec7d1c40df9a835b503d368d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.124758 1158 raft_consensus.cc:2270] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.125183 1158 ts_tablet_manager.cc:1905] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.126089 914 raft_consensus.cc:1273] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:52.126111 1155 consensus_queue.cc:260] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.126703 2049 consensus_queue.cc:1035] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:52.127296 1155 ts_tablet_manager.cc:1428] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.052s user 0.017s sys 0.001s
I20250629 01:51:52.127537 1155 tablet_bootstrap.cc:492] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.128772 2052 raft_consensus.cc:491] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.128875 2052 raft_consensus.cc:513] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.129333 2052 leader_election.cc:290] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.129547 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cf553b291854a6f9e7ed78b5db5f18c" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.129951 712 leader_election.cc:343] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.130751 1155 tablet_bootstrap.cc:654] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.132527 1158 ts_tablet_manager.cc:1918] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:52.132628 1158 log.cc:1199] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/bedbb041e4094d8399de625badd971bb
I20250629 01:51:52.132954 1158 ts_tablet_manager.cc:1939] T bedbb041e4094d8399de625badd971bb P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:52.133177 850 leader_election.cc:343] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 26d85fd3d3dd4b07a812c019e1a5c748
I20250629 01:51:52.133258 850 leader_election.cc:304] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.133582 1915 raft_consensus.cc:2747] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.134073 850 leader_election.cc:304] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.137147 1155 tablet_bootstrap.cc:492] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.137742 1155 ts_tablet_manager.cc:1397] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250629 01:51:52.138265 1155 raft_consensus.cc:357] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.138393 1155 raft_consensus.cc:383] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.138439 1155 raft_consensus.cc:738] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.139479 1155 consensus_queue.cc:260] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.140228 1155 ts_tablet_manager.cc:1428] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:52.140743 1155 tablet_bootstrap.cc:492] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.142983 1155 tablet_bootstrap.cc:654] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.144025 1156 ts_tablet_manager.cc:1918] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.144273 1159 tablet_bootstrap.cc:654] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.148831 1156 log.cc:1199] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0827ec622ff04ca6b2fec50e40b9e1de
I20250629 01:51:52.149466 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet aef865f51a6b4191b2ab453c15094cc1
I20250629 01:51:52.150194 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet dbfcb9233ace429db5daa593c2a1dcbe
I20250629 01:51:52.150729 449 catalog_manager.cc:6652] ReplaceTablet: tablet dbfcb9233ace429db5daa593c2a1dcbe deleted and replaced by tablet cc65674750c64114b999e0e292a851f9
I20250629 01:51:52.151091 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c5e9646c3ad04ed580f1d3402010dd38
I20250629 01:51:52.151499 455 catalog_manager.cc:6652] ReplaceTablet: tablet c5e9646c3ad04ed580f1d3402010dd38 deleted and replaced by tablet dc1b24606e6f4decb30364fa4fafc147
I20250629 01:51:52.154023 1915 raft_consensus.cc:2747] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.154129 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet da93c730451d42bc8df1e582fb3af818
I20250629 01:51:52.149515 1156 ts_tablet_manager.cc:1939] T 0827ec622ff04ca6b2fec50e40b9e1de P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.149847 1155 tablet_bootstrap.cc:492] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.156395 1155 ts_tablet_manager.cc:1397] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250629 01:51:52.157119 1155 raft_consensus.cc:357] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.157342 1155 raft_consensus.cc:383] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.157436 1155 raft_consensus.cc:738] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.157701 1155 consensus_queue.cc:260] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.158252 1155 ts_tablet_manager.cc:1428] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:52.158553 1155 tablet_bootstrap.cc:492] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.160004 1155 tablet_bootstrap.cc:654] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.162047 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_62271eae9afc4deca5dfc76ed3c291c2" } new_table_name: "default.Table_dcbcbed65c4445e7843620142b602552" modify_external_catalogs: true
I20250629 01:51:52.164561 2078 tablet.cc:1722] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:52.166769 1156 tablet_replica.cc:331] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.166988 1156 raft_consensus.cc:2241] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:52.167107 2079 tablet.cc:1722] T 399f460962134ee0b018e9cb8f5d6d2b P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:52.167296 1156 raft_consensus.cc:2270] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.167858 1156 ts_tablet_manager.cc:1905] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.173202 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4bca012737544ca1839fbb5ab17332b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.173743 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bca012737544ca1839fbb5ab17332b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.175029 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b423766fabef4323aaa0a999696efb58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.175455 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b423766fabef4323aaa0a999696efb58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.175593 1155 tablet_bootstrap.cc:492] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.175705 1155 ts_tablet_manager.cc:1397] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250629 01:51:52.176242 1155 raft_consensus.cc:357] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.176367 1155 raft_consensus.cc:383] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.176401 1155 raft_consensus.cc:738] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.176577 1155 consensus_queue.cc:260] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.176644 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_97e9884737fa43fe910d121339257f05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.177059 1155 ts_tablet_manager.cc:1428] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20250629 01:51:52.177201 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97e9884737fa43fe910d121339257f05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.177290 1155 tablet_bootstrap.cc:492] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.178231 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.178342 1155 tablet_bootstrap.cc:654] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:52.178786 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.179854 2068 sys_catalog.cc:263] Verifying existing consensus state
I20250629 01:51:52.188830 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6cbce56fa7a54b0eb7109265617d47eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.189426 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cbce56fa7a54b0eb7109265617d47eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:52.194936 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 52884.4us
I20250629 01:51:52.197873 743 tablet_service.cc:1515] Processing DeleteTablet for tablet aef865f51a6b4191b2ab453c15094cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.198916 743 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfcb9233ace429db5daa593c2a1dcbe with delete_type TABLET_DATA_DELETED (replaced by tablet cc65674750c64114b999e0e292a851f9) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:52.203016 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 71678.4us
I20250629 01:51:52.206444 878 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfcb9233ace429db5daa593c2a1dcbe with delete_type TABLET_DATA_DELETED (replaced by tablet cc65674750c64114b999e0e292a851f9) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.206805 889 tablet_service.cc:1515] Processing DeleteTablet for tablet da93c730451d42bc8df1e582fb3af818 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.206943 877 tablet_service.cc:1515] Processing DeleteTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b24606e6f4decb30364fa4fafc147) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.206962 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b24606e6f4decb30364fa4fafc147) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.208328 447 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 399f460962134ee0b018e9cb8f5d6d2b (table default.Table_8b19cb4429164a77b7cd89742caf4afa [id=282409bfc86b422ba299a651ae80476f]). Expected version 1 got 0
I20250629 01:51:52.209885 743 tablet_service.cc:1515] Processing DeleteTablet for tablet da93c730451d42bc8df1e582fb3af818 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.210530 1157 ts_tablet_manager.cc:1918] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.213512 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0f650c0d6e344c3883bd177ecd2a699f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.214038 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f650c0d6e344c3883bd177ecd2a699f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.214597 447 catalog_manager.cc:5582] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:52.215410 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:52.215649 447 catalog_manager.cc:5582] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:52.215968 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0247394d7c3c4941a7ef5ad5fa572566" } new_table_name: "default.Table_f52b1033085542258b846453f2b91f3d" modify_external_catalogs: true
I20250629 01:51:52.217566 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3610a48595354365bc24c326de4fc485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.218096 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3610a48595354365bc24c326de4fc485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.231014 1861 raft_consensus.cc:491] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.231130 1861 raft_consensus.cc:513] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.231632 1861 leader_election.cc:290] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.232069 743 tablet_service.cc:1515] Processing DeleteTablet for tablet ea00b4559e03410885180286b5e9fd10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.232551 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24d0e668c49ac825b2578849ab2a3) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.232990 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 81b32bac5a2144d1943285f5f446d53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.233423 743 tablet_service.cc:1515] Processing DeleteTablet for tablet df66984acfa44a99bf69096aff6d49ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.233840 743 tablet_service.cc:1515] Processing DeleteTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 with delete_type TABLET_DATA_DELETED (replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.234253 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.234695 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.237557 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0827ec622ff04ca6b2fec50e40b9e1de (table default.Table_8c1fbbaabe004466858cce35b18194b1 [id=700ef08e094b44369154076e65d9ea94]) successfully deleted
I20250629 01:51:52.238301 743 tablet_service.cc:1468] Processing CreateTablet for tablet 4416bbec7d1c40df9a835b503d368d4c (DEFAULT_TABLE table=default.Table_9396b8200cc74c71b35de340567589a4 [id=09be72ea10df4751b36fda893f0b2406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.239068 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 196f1bd233ab4399a9d22a55b8bc2635 (table default.Table_821e341e379740a59615270511038004 [id=910c490dcc294f58b93e5b6c4846d263]) successfully deleted
I20250629 01:51:52.239111 738 tablet_service.cc:1468] Processing CreateTablet for tablet ac24b3d316ef4450b4aecd531cfd801a (DEFAULT_TABLE table=default.Table_7dde0c66922743c1b227bc48f1adbd55 [id=468e184c6b3d41a593f330225dc84cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.239256 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a485e0e6e2f04aa6bd67a62f6d30be26 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:51:52.239444 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 34172d05ae40417494cd8f67f4cb981e (table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]) successfully deleted
I20250629 01:51:52.239600 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8dd084685cc0407da9e12b0a4df386e9 (table default.Table_875984d711bd4f7da73063fdc6fba264 [id=af75db4dc824485997f324795f65607b]) successfully deleted
W20250629 01:51:52.239804 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet faca0e904a04456d9aead23abd44e00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faca0e904a04456d9aead23abd44e00a already in progress: creating tablet
I20250629 01:51:52.239962 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]) successfully deleted
I20250629 01:51:52.242439 1156 ts_tablet_manager.cc:1918] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250629 01:51:52.242543 1156 log.cc:1199] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/57f7f71ab5e54d249141c89d9e806111
I20250629 01:51:52.242873 1156 ts_tablet_manager.cc:1939] T 57f7f71ab5e54d249141c89d9e806111 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.243708 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Bootstrap starting.
I20250629 01:51:52.244019 1154 tablet_bootstrap.cc:492] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.244177 1154 ts_tablet_manager.cc:1397] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.226s user 0.056s sys 0.103s
I20250629 01:51:52.244942 1154 raft_consensus.cc:357] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.245126 1154 raft_consensus.cc:383] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.245187 1154 raft_consensus.cc:738] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.245389 1154 consensus_queue.cc:260] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.245882 1154 ts_tablet_manager.cc:1428] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:52.246174 1154 tablet_bootstrap.cc:492] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.247821 1157 log.cc:1199] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4ed143377709465d9ae5df7c2ec3ca47
I20250629 01:51:52.248140 1157 ts_tablet_manager.cc:1939] T 4ed143377709465d9ae5df7c2ec3ca47 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:52.212484 895 tablet_service.cc:1468] Processing CreateTablet for tablet ac24b3d316ef4450b4aecd531cfd801a (DEFAULT_TABLE table=default.Table_7dde0c66922743c1b227bc48f1adbd55 [id=468e184c6b3d41a593f330225dc84cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.219316 881 tablet_service.cc:1515] Processing DeleteTablet for tablet aef865f51a6b4191b2ab453c15094cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.248492 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac24b3d316ef4450b4aecd531cfd801a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.248478 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 1336aea8f6b1436a971766d92d3c1197 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.248826 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 2049e7465da4404a86dbaf5a77adcda6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.249125 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet aef865f51a6b4191b2ab453c15094cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.249387 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet dbfcb9233ace429db5daa593c2a1dcbe with delete_type TABLET_DATA_DELETED (replaced by tablet cc65674750c64114b999e0e292a851f9) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.249634 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b24606e6f4decb30364fa4fafc147) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.249881 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet da93c730451d42bc8df1e582fb3af818 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.250315 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 4416bbec7d1c40df9a835b503d368d4c (DEFAULT_TABLE table=default.Table_9396b8200cc74c71b35de340567589a4 [id=09be72ea10df4751b36fda893f0b2406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.251070 1048 raft_consensus.cc:1273] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:52.251874 710 leader_election.cc:343] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 47d4287317ec4788bd96ad174c31748d
I20250629 01:51:52.252000 710 leader_election.cc:304] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.232066 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81b32bac5a2144d1943285f5f446d53d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.252240 1861 raft_consensus.cc:2747] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.252471 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ac24b3d316ef4450b4aecd531cfd801a (DEFAULT_TABLE table=default.Table_7dde0c66922743c1b227bc48f1adbd55 [id=468e184c6b3d41a593f330225dc84cac]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:52.252543 710 leader_election.cc:343] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:52.252648 710 leader_election.cc:304] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:52.252810 712 leader_election.cc:343] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.252833 1861 raft_consensus.cc:2747] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.253151 1861 consensus_queue.cc:1035] T 399f460962134ee0b018e9cb8f5d6d2b P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:52.253475 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4416bbec7d1c40df9a835b503d368d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.254312 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac24b3d316ef4450b4aecd531cfd801a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:52.259663 710 leader_election.cc:343] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:52.259758 710 leader_election.cc:304] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.260037 2049 raft_consensus.cc:2747] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.273672 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4416bbec7d1c40df9a835b503d368d4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.276330 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac24b3d316ef4450b4aecd531cfd801a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.276654 881 tablet_service.cc:1468] Processing CreateTablet for tablet fb114960c8124a16a3fc123e216ae881 (DEFAULT_TABLE table=default.Table_c2b04ca6a212457b94cd696da4139933 [id=b918a6f73e1f4d0d8f1cce53daaef9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.276948 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb114960c8124a16a3fc123e216ae881. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.276942 890 tablet_service.cc:1468] Processing CreateTablet for tablet df4c253b0c1d41259686c43e202a615f (DEFAULT_TABLE table=default.Table_d499c5244b63442cbb0110787b2c209d [id=b88a2362b5114d78815e8eec1148a322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.277235 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df4c253b0c1d41259686c43e202a615f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.277258 877 tablet_service.cc:1468] Processing CreateTablet for tablet ee1baa976b5e47e490a0fd561b37f83d (DEFAULT_TABLE table=default.Table_4db6832e616444be90ea5b21d5dace4f [id=af5bb41250204bca893542f1faab2987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.277529 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee1baa976b5e47e490a0fd561b37f83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.280298 888 tablet_service.cc:1468] Processing CreateTablet for tablet a5ebcf8ab6a14fc68f07a230249b0490 (DEFAULT_TABLE table=default.Table_3adc76548b5247bd99b926acdb4bea43 [id=247eff20b90f4b7f943e1b86c7e7aa42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.280568 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5ebcf8ab6a14fc68f07a230249b0490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.284288 896 tablet_service.cc:1468] Processing CreateTablet for tablet ccd0955a6476413680722e092b25bcf4 (DEFAULT_TABLE table=default.Table_b1a6331211334c548b6c97e62280bdff [id=ec76e6d8a2bc42df887c783ced596a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.284415 878 tablet_service.cc:1468] Processing CreateTablet for tablet 4af33b24ea404b1ab2608ea2e12b52a8 (DEFAULT_TABLE table=default.Table_e998111141ca474dbff482e945a67465 [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.284547 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd0955a6476413680722e092b25bcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.284667 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af33b24ea404b1ab2608ea2e12b52a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.288679 2049 raft_consensus.cc:491] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.288800 2049 raft_consensus.cc:513] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.289387 2049 leader_election.cc:290] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.292436 879 tablet_service.cc:1468] Processing CreateTablet for tablet 2a876b3db3904a4297b5b13a9325b1c3 (DEFAULT_TABLE table=default.Table_3bd1be2b90da48c7aed1e37d5508c9c1 [id=64c463ff92e64f16818a19e1a71083eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.292518 743 tablet_service.cc:1468] Processing CreateTablet for tablet 3acb79ba0de24579b14f41ce6670e956 (DEFAULT_TABLE table=default.Table_285e96b1ff674385850a3476dd050440 [id=c1df974c07ce4ede964f52e9ac91dd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.292806 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3acb79ba0de24579b14f41ce6670e956. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.292836 883 tablet_service.cc:1468] Processing CreateTablet for tablet 16342cedfef24b7e9082e9f1f9f32b5d (DEFAULT_TABLE table=default.Table_14435d8c8f2846e393084f1fa1f7f303 [id=1e0b2fe7ebe949cda774beb83024d628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.293696 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "432680cae61946af9859760fe23110ec" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.294080 712 leader_election.cc:343] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.284293 889 tablet_service.cc:1468] Processing CreateTablet for tablet 971dd8a2475043e3bac16b4c40b7c973 (DEFAULT_TABLE table=default.Table_f2c10fb36e7d4a3db070958ca3ffeeb7 [id=7f2665788d314dcb8244df8d9c03b570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.296579 1159 tablet_bootstrap.cc:492] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.296689 1159 ts_tablet_manager.cc:1397] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.202s user 0.002s sys 0.001s
I20250629 01:51:52.297336 1159 raft_consensus.cc:357] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.297679 1159 raft_consensus.cc:383] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.297729 1159 raft_consensus.cc:738] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.297915 1159 consensus_queue.cc:260] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.298106 738 tablet_service.cc:1468] Processing CreateTablet for tablet fb114960c8124a16a3fc123e216ae881 (DEFAULT_TABLE table=default.Table_c2b04ca6a212457b94cd696da4139933 [id=b918a6f73e1f4d0d8f1cce53daaef9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.298374 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb114960c8124a16a3fc123e216ae881. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.299415 1159 ts_tablet_manager.cc:1428] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:52.299628 1159 tablet_bootstrap.cc:492] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.301079 1159 tablet_bootstrap.cc:654] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:52.301836 710 leader_election.cc:343] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 432680cae61946af9859760fe23110ec
I20250629 01:51:52.301921 710 leader_election.cc:304] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.302184 2049 raft_consensus.cc:2747] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.303911 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 3acb79ba0de24579b14f41ce6670e956 (DEFAULT_TABLE table=default.Table_285e96b1ff674385850a3476dd050440 [id=c1df974c07ce4ede964f52e9ac91dd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.304286 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3acb79ba0de24579b14f41ce6670e956. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.305687 738 tablet_service.cc:1468] Processing CreateTablet for tablet ee1baa976b5e47e490a0fd561b37f83d (DEFAULT_TABLE table=default.Table_4db6832e616444be90ea5b21d5dace4f [id=af5bb41250204bca893542f1faab2987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.305998 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee1baa976b5e47e490a0fd561b37f83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.309454 755 tablet_service.cc:1468] Processing CreateTablet for tablet 971dd8a2475043e3bac16b4c40b7c973 (DEFAULT_TABLE table=default.Table_f2c10fb36e7d4a3db070958ca3ffeeb7 [id=7f2665788d314dcb8244df8d9c03b570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.309742 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971dd8a2475043e3bac16b4c40b7c973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.309715 750 tablet_service.cc:1468] Processing CreateTablet for tablet df4c253b0c1d41259686c43e202a615f (DEFAULT_TABLE table=default.Table_d499c5244b63442cbb0110787b2c209d [id=b88a2362b5114d78815e8eec1148a322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.309984 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df4c253b0c1d41259686c43e202a615f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.309995 1031 tablet_service.cc:1468] Processing CreateTablet for tablet fb114960c8124a16a3fc123e216ae881 (DEFAULT_TABLE table=default.Table_c2b04ca6a212457b94cd696da4139933 [id=b918a6f73e1f4d0d8f1cce53daaef9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.310161 749 tablet_service.cc:1468] Processing CreateTablet for tablet 4af33b24ea404b1ab2608ea2e12b52a8 (DEFAULT_TABLE table=default.Table_e998111141ca474dbff482e945a67465 [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.310250 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb114960c8124a16a3fc123e216ae881. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.310402 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af33b24ea404b1ab2608ea2e12b52a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.311290 2049 raft_consensus.cc:491] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.311383 2049 raft_consensus.cc:513] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.311797 2049 leader_election.cc:290] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.312605 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541728ff8dc3471ba90784b4a769c8e1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.312987 712 leader_election.cc:343] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.313045 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a876b3db3904a4297b5b13a9325b1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.313514 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541728ff8dc3471ba90784b4a769c8e1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:52.313645 1048 raft_consensus.cc:2466] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:52.313877 710 leader_election.cc:304] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:52.314081 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 4af33b24ea404b1ab2608ea2e12b52a8 (DEFAULT_TABLE table=default.Table_e998111141ca474dbff482e945a67465 [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.314157 2049 raft_consensus.cc:2802] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:52.314209 2049 raft_consensus.cc:491] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:52.314253 2049 raft_consensus.cc:3058] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.314258 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:52.314347 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af33b24ea404b1ab2608ea2e12b52a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.316315 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 971dd8a2475043e3bac16b4c40b7c973 (DEFAULT_TABLE table=default.Table_f2c10fb36e7d4a3db070958ca3ffeeb7 [id=7f2665788d314dcb8244df8d9c03b570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.316423 2049 raft_consensus.cc:513] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.316567 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971dd8a2475043e3bac16b4c40b7c973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.316900 2049 leader_election.cc:290] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.313887 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:52.317371 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541728ff8dc3471ba90784b4a769c8e1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:52.317497 1048 raft_consensus.cc:3058] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.317759 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ee1baa976b5e47e490a0fd561b37f83d (DEFAULT_TABLE table=default.Table_4db6832e616444be90ea5b21d5dace4f [id=af5bb41250204bca893542f1faab2987]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.318003 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee1baa976b5e47e490a0fd561b37f83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.319105 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "541728ff8dc3471ba90784b4a769c8e1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:52.319550 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16342cedfef24b7e9082e9f1f9f32b5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:52.319514 712 leader_election.cc:343] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.321007 1159 tablet_bootstrap.cc:492] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.321092 1159 ts_tablet_manager.cc:1397] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250629 01:51:52.321663 1159 raft_consensus.cc:357] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.321775 1159 raft_consensus.cc:383] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.321810 1159 raft_consensus.cc:738] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.322037 1159 consensus_queue.cc:260] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.323549 1159 ts_tablet_manager.cc:1428] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:52.323621 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971dd8a2475043e3bac16b4c40b7c973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.323760 1159 tablet_bootstrap.cc:492] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.325174 1159 tablet_bootstrap.cc:654] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.325922 1048 raft_consensus.cc:2466] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:52.326184 710 leader_election.cc:304] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:52.326496 2049 raft_consensus.cc:2802] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:52.326588 2049 raft_consensus.cc:695] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:52.326774 2049 consensus_queue.cc:237] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.333915 1155 tablet_bootstrap.cc:492] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.333935 1029 tablet_service.cc:1468] Processing CreateTablet for tablet df4c253b0c1d41259686c43e202a615f (DEFAULT_TABLE table=default.Table_d499c5244b63442cbb0110787b2c209d [id=b88a2362b5114d78815e8eec1148a322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.334020 1155 ts_tablet_manager.cc:1397] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.049s
I20250629 01:51:52.334013 750 tablet_service.cc:1468] Processing CreateTablet for tablet a5ebcf8ab6a14fc68f07a230249b0490 (DEFAULT_TABLE table=default.Table_3adc76548b5247bd99b926acdb4bea43 [id=247eff20b90f4b7f943e1b86c7e7aa42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.334177 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df4c253b0c1d41259686c43e202a615f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.334265 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5ebcf8ab6a14fc68f07a230249b0490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.334565 1155 raft_consensus.cc:357] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.334685 1155 raft_consensus.cc:383] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.334723 1155 raft_consensus.cc:738] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.334884 1155 consensus_queue.cc:260] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.335232 1155 ts_tablet_manager.cc:1428] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:52.335423 1155 tablet_bootstrap.cc:492] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.336504 1155 tablet_bootstrap.cc:654] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.338595 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:52.340880 712 consensus_peers.cc:489] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:52.341130 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a5ebcf8ab6a14fc68f07a230249b0490 (DEFAULT_TABLE table=default.Table_3adc76548b5247bd99b926acdb4bea43 [id=247eff20b90f4b7f943e1b86c7e7aa42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.341315 1159 tablet_bootstrap.cc:492] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.341434 1159 ts_tablet_manager.cc:1397] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250629 01:51:52.341658 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5ebcf8ab6a14fc68f07a230249b0490. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.344794 1155 tablet_bootstrap.cc:492] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.344900 1155 ts_tablet_manager.cc:1397] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250629 01:51:52.345247 1159 raft_consensus.cc:357] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.345428 1155 raft_consensus.cc:357] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.345552 1155 raft_consensus.cc:383] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.345600 1155 raft_consensus.cc:738] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.345790 1155 consensus_queue.cc:260] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.346139 1155 ts_tablet_manager.cc:1428] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:52.346344 1155 tablet_bootstrap.cc:492] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.345373 1159 raft_consensus.cc:383] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.346992 1159 raft_consensus.cc:738] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.347337 1159 consensus_queue.cc:260] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.347681 1155 tablet_bootstrap.cc:654] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.348474 749 tablet_service.cc:1468] Processing CreateTablet for tablet ccd0955a6476413680722e092b25bcf4 (DEFAULT_TABLE table=default.Table_b1a6331211334c548b6c97e62280bdff [id=ec76e6d8a2bc42df887c783ced596a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.348778 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd0955a6476413680722e092b25bcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.348891 1156 tablet_replica.cc:331] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.349107 1156 raft_consensus.cc:2241] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.349269 1156 raft_consensus.cc:2270] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.349352 1159 ts_tablet_manager.cc:1428] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250629 01:51:52.349767 1156 ts_tablet_manager.cc:1905] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.349927 1861 raft_consensus.cc:491] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.350019 1861 raft_consensus.cc:513] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.350485 1861 leader_election.cc:290] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:52.351130 710 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: fca257d6cb614110948e1045dee44bdc
I20250629 01:51:52.355935 1155 tablet_bootstrap.cc:492] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.356024 1155 ts_tablet_manager.cc:1397] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250629 01:51:52.356505 1155 raft_consensus.cc:357] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.356624 1155 raft_consensus.cc:383] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.356679 1155 raft_consensus.cc:738] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.356936 1155 consensus_queue.cc:260] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.357297 1155 ts_tablet_manager.cc:1428] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:52.357483 1155 tablet_bootstrap.cc:492] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.358680 1155 tablet_bootstrap.cc:654] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.361342 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ccd0955a6476413680722e092b25bcf4 (DEFAULT_TABLE table=default.Table_b1a6331211334c548b6c97e62280bdff [id=ec76e6d8a2bc42df887c783ced596a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.361683 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccd0955a6476413680722e092b25bcf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.365643 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 2a876b3db3904a4297b5b13a9325b1c3 (DEFAULT_TABLE table=default.Table_3bd1be2b90da48c7aed1e37d5508c9c1 [id=64c463ff92e64f16818a19e1a71083eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.366035 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a876b3db3904a4297b5b13a9325b1c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:52.368252 712 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: fca257d6cb614110948e1045dee44bdc
I20250629 01:51:52.368341 712 leader_election.cc:304] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.368616 1861 raft_consensus.cc:2747] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:52.375890 1861 raft_consensus.cc:491] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.375984 1861 raft_consensus.cc:513] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.376449 1861 leader_election.cc:290] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.376840 1154 tablet_bootstrap.cc:654] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.376874 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35990d206ba64159ab05ab7b7b19d064" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.376968 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 16342cedfef24b7e9082e9f1f9f32b5d (DEFAULT_TABLE table=default.Table_14435d8c8f2846e393084f1fa1f7f303 [id=1e0b2fe7ebe949cda774beb83024d628]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.377244 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16342cedfef24b7e9082e9f1f9f32b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.377559 895 tablet_service.cc:1468] Processing CreateTablet for tablet 3acb79ba0de24579b14f41ce6670e956 (DEFAULT_TABLE table=default.Table_285e96b1ff674385850a3476dd050440 [id=c1df974c07ce4ede964f52e9ac91dd45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.377825 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3acb79ba0de24579b14f41ce6670e956. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.378278 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_ab9e746444a24266aca5439c51b30a27 [id=60094a0a7c134476a2a4e44d8555c5d6]) successfully deleted
W20250629 01:51:52.378731 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1336aea8f6b1436a971766d92d3c1197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1336aea8f6b1436a971766d92d3c1197 already in progress: creating tablet
W20250629 01:51:52.378949 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2049e7465da4404a86dbaf5a77adcda6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2049e7465da4404a86dbaf5a77adcda6 already in progress: creating tablet
W20250629 01:51:52.379117 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet aef865f51a6b4191b2ab453c15094cc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef865f51a6b4191b2ab453c15094cc1 already in progress: creating tablet
W20250629 01:51:52.379308 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet dbfcb9233ace429db5daa593c2a1dcbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfcb9233ace429db5daa593c2a1dcbe already in progress: creating tablet
W20250629 01:51:52.379498 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c5e9646c3ad04ed580f1d3402010dd38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5e9646c3ad04ed580f1d3402010dd38 already in progress: creating tablet
W20250629 01:51:52.379695 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet da93c730451d42bc8df1e582fb3af818 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da93c730451d42bc8df1e582fb3af818 already in progress: creating tablet
W20250629 01:51:52.380432 712 leader_election.cc:343] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:52.381698 710 leader_election.cc:343] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 35990d206ba64159ab05ab7b7b19d064
I20250629 01:51:52.381793 710 leader_election.cc:304] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.382138 2052 raft_consensus.cc:2747] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.385288 755 tablet_service.cc:1468] Processing CreateTablet for tablet 2a876b3db3904a4297b5b13a9325b1c3 (DEFAULT_TABLE table=default.Table_3bd1be2b90da48c7aed1e37d5508c9c1 [id=64c463ff92e64f16818a19e1a71083eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.385612 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a876b3db3904a4297b5b13a9325b1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.387284 1915 raft_consensus.cc:491] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.387394 1915 raft_consensus.cc:513] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.387814 1915 leader_election.cc:290] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.388201 1154 tablet_bootstrap.cc:492] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.388330 1154 ts_tablet_manager.cc:1397] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.142s user 0.002s sys 0.001s
I20250629 01:51:52.388429 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:52.388996 1159 tablet_bootstrap.cc:492] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.390048 1861 raft_consensus.cc:491] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.390136 1861 raft_consensus.cc:513] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.390446 1159 tablet_bootstrap.cc:654] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.390566 1861 leader_election.cc:290] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.391027 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf4b52fae47f45ad9d77b0a97a854860" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:52.391328 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34172d05ae40417494cd8f67f4cb981e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.391577 850 leader_election.cc:304] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:52.391932 712 leader_election.cc:343] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.392148 1915 raft_consensus.cc:2747] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:52.392997 710 leader_election.cc:343] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:52.393071 710 leader_election.cc:304] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.393381 1861 raft_consensus.cc:2747] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.395246 1156 ts_tablet_manager.cc:1918] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.395341 1156 log.cc:1199] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b9b680859a1a4c56afe33f5b1b3c0793
I20250629 01:51:52.395620 1156 ts_tablet_manager.cc:1939] T b9b680859a1a4c56afe33f5b1b3c0793 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.397383 1154 raft_consensus.cc:357] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.397579 1154 raft_consensus.cc:383] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.397641 1154 raft_consensus.cc:738] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.398981 1155 tablet_bootstrap.cc:492] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.399091 1155 ts_tablet_manager.cc:1397] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.020s
I20250629 01:51:52.399674 1155 raft_consensus.cc:357] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.399804 1155 raft_consensus.cc:383] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.399843 1155 raft_consensus.cc:738] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.399798 1154 consensus_queue.cc:260] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.400022 1155 consensus_queue.cc:260] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.400463 1155 ts_tablet_manager.cc:1428] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:52.400575 1154 ts_tablet_manager.cc:1428] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250629 01:51:52.400676 1155 tablet_bootstrap.cc:492] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.400771 1154 tablet_bootstrap.cc:492] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.401872 1154 tablet_bootstrap.cc:654] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.401880 1155 tablet_bootstrap.cc:654] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.406003 1157 tablet_replica.cc:331] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:52.406162 1157 raft_consensus.cc:2241] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.406277 1157 raft_consensus.cc:2270] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.406661 1157 ts_tablet_manager.cc:1905] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.407656 1915 raft_consensus.cc:491] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.407748 1915 raft_consensus.cc:513] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.408370 1915 leader_election.cc:290] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:52.409409 850 leader_election.cc:343] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a90b206d58d04acdb1a8cb0c45d68c31
W20250629 01:51:52.409538 850 leader_election.cc:343] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: a90b206d58d04acdb1a8cb0c45d68c31
I20250629 01:51:52.409587 850 leader_election.cc:304] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.409883 1915 raft_consensus.cc:2747] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.410857 883 tablet_service.cc:1468] Processing CreateTablet for tablet eb376259f6cd490b87b52cfc1f8e6275 (DEFAULT_TABLE table=default.Table_4bd3cf3371774492beb8e01eec5349fc [id=f8941cf863884040b9942d7d36e01fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.411911 2068 tablet_bootstrap.cc:700] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Time spent opening tablet: real 0.159s user 0.001s sys 0.000s
I20250629 01:51:52.410858 755 tablet_service.cc:1468] Processing CreateTablet for tablet ac2270c5abd4456890378c7f82b5f663 (DEFAULT_TABLE table=default.Table_899f5cb3f36f46dd8e4c809803070720 [id=2cc28d7212814bbaba48d136fdf5ed7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.412747 877 tablet_service.cc:1468] Processing CreateTablet for tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2 (DEFAULT_TABLE table=default.Table_c951e1bffffa4ed7b7fcff1b87d9ee23 [id=3955cdcd5c2449dba48d27b07ea2ed71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.415154 895 tablet_service.cc:1468] Processing CreateTablet for tablet ac2270c5abd4456890378c7f82b5f663 (DEFAULT_TABLE table=default.Table_899f5cb3f36f46dd8e4c809803070720 [id=2cc28d7212814bbaba48d136fdf5ed7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.414866 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2270c5abd4456890378c7f82b5f663. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.415984 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ac2270c5abd4456890378c7f82b5f663 (DEFAULT_TABLE table=default.Table_899f5cb3f36f46dd8e4c809803070720 [id=2cc28d7212814bbaba48d136fdf5ed7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.416579 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2270c5abd4456890378c7f82b5f663. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.419719 1159 tablet_bootstrap.cc:492] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.419860 1159 ts_tablet_manager.cc:1397] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.001s
I20250629 01:51:52.420490 1159 raft_consensus.cc:357] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.420609 1159 raft_consensus.cc:383] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.420646 1159 raft_consensus.cc:738] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.420806 2095 tablet.cc:1722] T 399f460962134ee0b018e9cb8f5d6d2b P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:52.421073 1159 consensus_queue.cc:260] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.421541 1159 ts_tablet_manager.cc:1428] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:52.422606 1159 tablet_bootstrap.cc:492] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.423312 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.426404 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb376259f6cd490b87b52cfc1f8e6275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.434002 1159 tablet_bootstrap.cc:654] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.436798 2066 raft_consensus.cc:491] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.416848 1155 tablet_bootstrap.cc:492] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.426522 743 tablet_service.cc:1468] Processing CreateTablet for tablet eb376259f6cd490b87b52cfc1f8e6275 (DEFAULT_TABLE table=default.Table_4bd3cf3371774492beb8e01eec5349fc [id=f8941cf863884040b9942d7d36e01fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.427227 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2 (DEFAULT_TABLE table=default.Table_c951e1bffffa4ed7b7fcff1b87d9ee23 [id=3955cdcd5c2449dba48d27b07ea2ed71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.436923 2066 raft_consensus.cc:513] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.437179 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.437390 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb376259f6cd490b87b52cfc1f8e6275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.437538 2066 leader_election.cc:290] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:52.438180 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:52.438486 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.438563 986 leader_election.cc:343] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:52.439071 988 leader_election.cc:343] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.439157 988 leader_election.cc:304] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.440062 2066 raft_consensus.cc:2747] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.443328 1155 ts_tablet_manager.cc:1397] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.009s
I20250629 01:51:52.444356 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2 (DEFAULT_TABLE table=default.Table_c951e1bffffa4ed7b7fcff1b87d9ee23 [id=3955cdcd5c2449dba48d27b07ea2ed71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.444741 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.445068 1019 tablet_service.cc:1468] Processing CreateTablet for tablet eb376259f6cd490b87b52cfc1f8e6275 (DEFAULT_TABLE table=default.Table_4bd3cf3371774492beb8e01eec5349fc [id=f8941cf863884040b9942d7d36e01fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.445401 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb376259f6cd490b87b52cfc1f8e6275. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.449872 1155 raft_consensus.cc:357] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.452316 1154 tablet_bootstrap.cc:492] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.452003 1155 raft_consensus.cc:383] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.452804 1154 ts_tablet_manager.cc:1397] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20250629 01:51:52.452708 1155 raft_consensus.cc:738] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.454113 1154 raft_consensus.cc:357] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.454291 1154 raft_consensus.cc:383] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.455498 1154 raft_consensus.cc:738] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.455044 1155 consensus_queue.cc:260] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.457022 1154 consensus_queue.cc:260] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.458812 1154 ts_tablet_manager.cc:1428] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:51:52.460392 1154 tablet_bootstrap.cc:492] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.461572 1154 tablet_bootstrap.cc:654] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.461412 1155 ts_tablet_manager.cc:1428] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20250629 01:51:52.464198 1915 raft_consensus.cc:491] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.464298 1915 raft_consensus.cc:513] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.464758 1915 leader_election.cc:290] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.464629 1155 tablet_bootstrap.cc:492] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:52.465677 850 leader_election.cc:343] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 9624117e97974a47ac2bfe525da6f405
I20250629 01:51:52.467283 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:52.467643 850 leader_election.cc:343] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9624117e97974a47ac2bfe525da6f405
I20250629 01:51:52.467737 850 leader_election.cc:304] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.468077 1915 raft_consensus.cc:2747] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.482471 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2270c5abd4456890378c7f82b5f663. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.488868 749 tablet_service.cc:1468] Processing CreateTablet for tablet 16342cedfef24b7e9082e9f1f9f32b5d (DEFAULT_TABLE table=default.Table_14435d8c8f2846e393084f1fa1f7f303 [id=1e0b2fe7ebe949cda774beb83024d628]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:52.489174 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2591c65d53449c8801778606f1f05f1 consensus state has no leader
I20250629 01:51:52.489251 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16342cedfef24b7e9082e9f1f9f32b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.495621 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4ed143377709465d9ae5df7c2ec3ca47 (table default.Table_ace97daf931b45ac9676c39144d2931f [id=1669d2f9d2b842bfb8fdbc03b965669a]) successfully deleted
I20250629 01:51:52.496538 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b9b680859a1a4c56afe33f5b1b3c0793 (table default.Table_30467a20b5b54ca7894c405b22705bc2 [id=66af0a365c3f40deac642473f9125b10]) successfully deleted
W20250629 01:51:52.496521 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 81083.4us
I20250629 01:51:52.495739 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e67aa47b96b74730985e975276f3745b" } modify_external_catalogs: true
I20250629 01:51:52.497514 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e67aa47b96b74730985e975276f3745b" } modify_external_catalogs: true
I20250629 01:51:52.498916 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f07aa6eb42b6478781eb5a65dea53212" } new_table_name: "default.Table_600f51cec79c492496ab746496a0ede3" modify_external_catalogs: true
I20250629 01:51:52.502821 1156 tablet_replica.cc:331] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.502996 1156 raft_consensus.cc:2241] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:52.503738 1156 raft_consensus.cc:2270] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.504319 1156 ts_tablet_manager.cc:1905] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.505441 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_04bf790258f64e6f87859f34119d5c4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.505880 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04bf790258f64e6f87859f34119d5c4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.506165 1157 ts_tablet_manager.cc:1918] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.506242 1157 log.cc:1199] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/26d85fd3d3dd4b07a812c019e1a5c748
I20250629 01:51:52.506490 1157 ts_tablet_manager.cc:1939] T 26d85fd3d3dd4b07a812c019e1a5c748 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:52.507119 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_415759429f02467f944d3c1a215d5966" } modify_external_catalogs: true
I20250629 01:51:52.507236 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_415759429f02467f944d3c1a215d5966" } modify_external_catalogs: true
W20250629 01:51:52.508735 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 51924.9us
I20250629 01:51:52.509150 1155 tablet_bootstrap.cc:654] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:52.509814 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc65674750c64114b999e0e292a851f9 has no consensus state
I20250629 01:51:52.510099 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca2bcac172414e6c9f264cb2202a76fa" } new_table_name: "default.Table_5b1107f5f7624470a33b0defd6eedd44" modify_external_catalogs: true
I20250629 01:51:52.511214 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_12a83e36ffff41fa944eefd75cdc2bf9" } modify_external_catalogs: true
I20250629 01:51:52.511322 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_12a83e36ffff41fa944eefd75cdc2bf9" } modify_external_catalogs: true
I20250629 01:51:52.512252 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5beabed0e3e44bfe825659e35faa85fa" } new_table_name: "default.Table_17373cbe989b4dfcb4c39ed0f708d216" modify_external_catalogs: true
I20250629 01:51:52.513162 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 814) took 1045 ms. Trace:
I20250629 01:51:52.513227 1157 rpcz_store.cc:276] 0629 01:51:51.467908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:51.468995 (+ 1087us) service_pool.cc:225] Handling call
0629 01:51:52.405970 (+936975us) ts_tablet_manager.cc:1145] Deleting tablet 26d85fd3d3dd4b07a812c019e1a5c748
0629 01:51:52.405976 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:52.503770 (+ 97794us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:52.506097 (+ 2327us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:52.513144 (+ 7047us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":936826}
I20250629 01:51:52.516202 1157 tablet_replica.cc:331] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:52.516335 1157 raft_consensus.cc:2241] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.516440 1157 raft_consensus.cc:2270] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.516891 1157 ts_tablet_manager.cc:1905] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.519721 1155 tablet_bootstrap.cc:492] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.519830 1155 ts_tablet_manager.cc:1397] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.004s
I20250629 01:51:52.520620 1155 raft_consensus.cc:357] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.520769 1155 raft_consensus.cc:383] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.520818 1155 raft_consensus.cc:738] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.520972 1155 consensus_queue.cc:260] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.521400 1155 ts_tablet_manager.cc:1428] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:52.521623 1155 tablet_bootstrap.cc:492] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.522930 1155 tablet_bootstrap.cc:654] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.524562 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4c419cdd9add4db7b041836405c4856b" } modify_external_catalogs: true
I20250629 01:51:52.524705 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4c419cdd9add4db7b041836405c4856b" } modify_external_catalogs: true
W20250629 01:51:52.526855 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet faca0e904a04456d9aead23abd44e00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faca0e904a04456d9aead23abd44e00a already in progress: creating tablet
I20250629 01:51:52.527750 1157 ts_tablet_manager.cc:1918] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.527827 1157 log.cc:1199] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a32636c293bc44858da7f29a515c467b
I20250629 01:51:52.528033 1157 ts_tablet_manager.cc:1939] T a32636c293bc44858da7f29a515c467b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:52.528267 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 815) took 1021 ms. Trace:
I20250629 01:51:52.528317 1157 rpcz_store.cc:276] 0629 01:51:51.506496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:51.507467 (+ 971us) service_pool.cc:225] Handling call
0629 01:51:52.516167 (+1008700us) ts_tablet_manager.cc:1145] Deleting tablet a32636c293bc44858da7f29a515c467b
0629 01:51:52.516174 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:52.525486 (+ 9312us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:52.527692 (+ 2206us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:52.528256 (+ 564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1008535}
I20250629 01:51:52.529937 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.530118 1158 tablet_replica.cc:331] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.530290 1158 raft_consensus.cc:2241] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.530407 1158 raft_consensus.cc:2270] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.530944 1158 ts_tablet_manager.cc:1905] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.531736 1050 raft_consensus.cc:1273] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:52.536161 1158 ts_tablet_manager.cc:1918] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.536314 1158 log.cc:1199] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/1ea681bd91e54e4898bcfb72bed6198f
I20250629 01:51:52.536631 1158 ts_tablet_manager.cc:1939] T 1ea681bd91e54e4898bcfb72bed6198f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.537971 1159 tablet_bootstrap.cc:492] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.538117 1159 ts_tablet_manager.cc:1397] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.018s
I20250629 01:51:52.540364 1159 raft_consensus.cc:357] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.540555 1159 raft_consensus.cc:383] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.540632 1159 raft_consensus.cc:738] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.540812 2049 consensus_queue.cc:1035] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:52.540845 1159 consensus_queue.cc:260] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.542337 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 816) took 1015 ms. Trace:
I20250629 01:51:52.542423 1157 rpcz_store.cc:276] 0629 01:51:51.527172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:51.527220 (+ 48us) service_pool.cc:225] Handling call
0629 01:51:52.542220 (+1015000us) ts_tablet_manager.cc:1145] Deleting tablet 352a8dbd7f164ca9b622fed30bc6beeb
0629 01:51:52.542230 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:52.542320 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014834}
I20250629 01:51:52.542771 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 817) took 1007 ms. Trace:
I20250629 01:51:52.542814 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:52.543006 712 consensus_peers.cc:489] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:52.542833 1157 rpcz_store.cc:276] 0629 01:51:51.535546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:51.535601 (+ 55us) service_pool.cc:225] Handling call
0629 01:51:52.542651 (+1007050us) ts_tablet_manager.cc:1145] Deleting tablet 7908afa2abaa4521b44f954e0790f986
0629 01:51:52.542668 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:52.542757 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1006905}
I20250629 01:51:52.544991 1157 tablet_replica.cc:331] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:52.545184 1157 raft_consensus.cc:2241] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.545305 1157 raft_consensus.cc:2270] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.545782 1157 ts_tablet_manager.cc:1905] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.551851 1159 ts_tablet_manager.cc:1428] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.014s user 0.000s sys 0.001s
I20250629 01:51:52.553174 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 26d85fd3d3dd4b07a812c019e1a5c748 (table default.Table_331d3b611f4f46748132e37fb1a8e872 [id=b9df439e13104b14be7c1b522b04ec13]) successfully deleted
I20250629 01:51:52.555188 1154 tablet_bootstrap.cc:492] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.554337 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a32636c293bc44858da7f29a515c467b (table default.Table_2fc84dd48d464ce682425fbd41d92b7e [id=64b627ee531b41ffa46d4bbd813c5b8c]) successfully deleted
I20250629 01:51:52.555311 1154 ts_tablet_manager.cc:1397] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.018s
W20250629 01:51:52.555603 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 352a8dbd7f164ca9b622fed30bc6beeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 352a8dbd7f164ca9b622fed30bc6beeb already in progress: creating tablet
I20250629 01:51:52.555966 1159 tablet_bootstrap.cc:492] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.555971 1154 raft_consensus.cc:357] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.556157 1154 raft_consensus.cc:383] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.556205 1154 raft_consensus.cc:738] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:52.556314 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7908afa2abaa4521b44f954e0790f986 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7908afa2abaa4521b44f954e0790f986 already in progress: creating tablet
I20250629 01:51:52.556371 1154 consensus_queue.cc:260] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:52.556524 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0467b153057f491e81681fa399a2a1a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0467b153057f491e81681fa399a2a1a9 already in progress: creating tablet
W20250629 01:51:52.556697 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a20245e23a9e4dff9bfb1e565dc38184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a20245e23a9e4dff9bfb1e565dc38184 already in progress: creating tablet
I20250629 01:51:52.556792 1154 ts_tablet_manager.cc:1428] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:52.557014 1154 tablet_bootstrap.cc:492] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.557520 1156 ts_tablet_manager.cc:1918] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:52.557672 1156 log.cc:1199] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/bedbb041e4094d8399de625badd971bb
I20250629 01:51:52.558028 1156 ts_tablet_manager.cc:1939] T bedbb041e4094d8399de625badd971bb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.558122 1159 tablet_bootstrap.cc:654] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.558470 1154 tablet_bootstrap.cc:654] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.562108 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]) successfully deleted
W20250629 01:51:52.562288 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1336aea8f6b1436a971766d92d3c1197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1336aea8f6b1436a971766d92d3c1197 already in progress: creating tablet
W20250629 01:51:52.562435 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2049e7465da4404a86dbaf5a77adcda6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2049e7465da4404a86dbaf5a77adcda6 already in progress: creating tablet
W20250629 01:51:52.562587 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet aef865f51a6b4191b2ab453c15094cc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef865f51a6b4191b2ab453c15094cc1 already in progress: creating tablet
I20250629 01:51:52.563144 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 1ea681bd91e54e4898bcfb72bed6198f (table default.Table_1a2a249712224a0fa06b23867c043cfa [id=2fbd2c29b63c488ba11a1ed546f38da4]) successfully deleted
I20250629 01:51:52.563427 1159 tablet_bootstrap.cc:492] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.564235 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ef95de81b1de41d29a5db6dcdd5dd270
I20250629 01:51:52.567943 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ef95de81b1de41d29a5db6dcdd5dd270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.568432 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet ef95de81b1de41d29a5db6dcdd5dd270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.570178 1155 tablet_bootstrap.cc:492] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.570286 1155 ts_tablet_manager.cc:1397] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.001s
W20250629 01:51:52.570640 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ef95de81b1de41d29a5db6dcdd5dd270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef95de81b1de41d29a5db6dcdd5dd270 already in progress: creating tablet
I20250629 01:51:52.570919 1155 raft_consensus.cc:357] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.571070 1155 raft_consensus.cc:383] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.571128 1155 raft_consensus.cc:738] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.571219 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0891a717f4824360be1c4fe4e773a511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.571287 1155 consensus_queue.cc:260] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.571637 1155 ts_tablet_manager.cc:1428] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250629 01:51:52.571695 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0891a717f4824360be1c4fe4e773a511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.571906 1155 tablet_bootstrap.cc:492] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.573329 1155 tablet_bootstrap.cc:654] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.567329 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ef95de81b1de41d29a5db6dcdd5dd270 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.576246 1159 ts_tablet_manager.cc:1397] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
I20250629 01:51:52.586483 2049 raft_consensus.cc:491] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.586586 2049 raft_consensus.cc:513] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.587040 2049 leader_election.cc:290] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.587877 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:52.588023 1049 raft_consensus.cc:2466] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:52.588459 710 leader_election.cc:304] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:52.588748 2049 raft_consensus.cc:2802] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:52.588812 2049 raft_consensus.cc:491] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:52.588872 2049 raft_consensus.cc:3058] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.589030 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.590098 1157 ts_tablet_manager.cc:1918] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.590438 1157 log.cc:1199] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/be8e0946a20340b38835c4d8daeaa7b8
I20250629 01:51:52.590880 1155 tablet_bootstrap.cc:492] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.591141 2049 raft_consensus.cc:513] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.591574 2066 raft_consensus.cc:491] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.591495 1155 ts_tablet_manager.cc:1397] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250629 01:51:52.591675 2066 raft_consensus.cc:513] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.592159 2066 leader_election.cc:290] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:52.592036 1157 ts_tablet_manager.cc:1939] T be8e0946a20340b38835c4d8daeaa7b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:52.592682 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5941dd6381041538be6c92858d6ba13" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.592844 712 leader_election.cc:343] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.592957 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5941dd6381041538be6c92858d6ba13" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:52.593241 988 leader_election.cc:343] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:52.593676 986 leader_election.cc:343] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.593760 986 leader_election.cc:304] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.593773 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet be8e0946a20340b38835c4d8daeaa7b8 (table default.Table_f2bc100d78a348c88ed48ed66870200b [id=59260e8695bb4417a47c087d392cbad7]) successfully deleted
I20250629 01:51:52.594086 2066 raft_consensus.cc:2747] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.594133 1157 tablet_replica.cc:331] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:52.594688 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:52.595017 1157 raft_consensus.cc:2241] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:52.595201 712 leader_election.cc:343] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.595772 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:52.595288 1155 raft_consensus.cc:357] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.595904 1049 raft_consensus.cc:3058] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.595793 1157 raft_consensus.cc:2270] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.596127 1155 raft_consensus.cc:383] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.596279 1155 raft_consensus.cc:738] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.598107 1049 raft_consensus.cc:2466] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:52.597256 1155 consensus_queue.cc:260] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.598557 710 leader_election.cc:304] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:52.598765 2049 leader_election.cc:290] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.598985 2049 raft_consensus.cc:2802] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:52.599115 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cc67f532875d4d49af4ed8a1673ec007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.599372 2049 raft_consensus.cc:695] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
W20250629 01:51:52.599556 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet caa63fe10ff9487a9fc16479285c5abd has no consensus state
W20250629 01:51:52.599610 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc67f532875d4d49af4ed8a1673ec007 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.599563 2049 consensus_queue.cc:237] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.599887 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8ffd172d7dc7411083dcbc1e695b647d" } modify_external_catalogs: true
I20250629 01:51:52.600021 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8ffd172d7dc7411083dcbc1e695b647d" } modify_external_catalogs: true
I20250629 01:51:52.599090 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5deb2a3d4b3a4a8d8acfbf4a31002495
I20250629 01:51:52.600893 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 12b85f220f3947ca8110f2db937e7b68 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:52.606141 2095 tablet.cc:1722] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:52.610915 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.576790 1159 raft_consensus.cc:357] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.611021 1159 raft_consensus.cc:383] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.611446 1159 raft_consensus.cc:738] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.611275 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.611735 1159 consensus_queue.cc:260] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:52.611981 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet dbfcb9233ace429db5daa593c2a1dcbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfcb9233ace429db5daa593c2a1dcbe already in progress: creating tablet
I20250629 01:51:52.612298 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.614254 1861 raft_consensus.cc:491] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.614360 1861 raft_consensus.cc:513] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.614813 1861 leader_election.cc:290] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.614861 1159 ts_tablet_manager.cc:1428] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250629 01:51:52.615056 1159 tablet_bootstrap.cc:492] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.616603 1156 tablet_replica.cc:331] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.616688 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "df66984acfa44a99bf69096aff6d49ab" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.616770 1156 raft_consensus.cc:2241] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.616878 1156 raft_consensus.cc:2270] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:52.617183 712 leader_election.cc:343] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.617611 2078 tablet.cc:1722] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:52.617779 1156 ts_tablet_manager.cc:1905] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:52.617918 710 leader_election.cc:343] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:52.617988 710 leader_election.cc:304] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.618263 1861 raft_consensus.cc:2747] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:52.618522 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c5e9646c3ad04ed580f1d3402010dd38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5e9646c3ad04ed580f1d3402010dd38 already in progress: creating tablet
W20250629 01:51:52.619858 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet da93c730451d42bc8df1e582fb3af818 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da93c730451d42bc8df1e582fb3af818 already in progress: creating tablet
W20250629 01:51:52.620066 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 already in progress: creating tablet
I20250629 01:51:52.621512 1154 tablet_bootstrap.cc:492] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.621634 1154 ts_tablet_manager.cc:1397] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.002s
I20250629 01:51:52.622231 1154 raft_consensus.cc:357] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.622360 1154 raft_consensus.cc:383] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.622406 1154 raft_consensus.cc:738] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.622560 1154 consensus_queue.cc:260] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.623083 1157 ts_tablet_manager.cc:1905] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.623327 1159 tablet_bootstrap.cc:654] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.624838 1155 ts_tablet_manager.cc:1428] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20250629 01:51:52.626097 1155 tablet_bootstrap.cc:492] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.628362 1154 ts_tablet_manager.cc:1428] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:52.629400 451 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet cd0964f6390c4dea9d49c7d0e23d9dcb
I20250629 01:51:52.629597 451 master_service.cc:948] ReplaceTablet: received request to replace tablet cf4b52fae47f45ad9d77b0a97a854860 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:52.630321 914 raft_consensus.cc:3058] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.631058 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c5d45854191247bf8172d0bbfd01b6db
I20250629 01:51:52.631075 1156 ts_tablet_manager.cc:1918] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.631151 1156 log.cc:1199] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ea00b4559e03410885180286b5e9fd10
I20250629 01:51:52.631397 1156 ts_tablet_manager.cc:1939] T ea00b4559e03410885180286b5e9fd10 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.631639 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 59cc1690878f40e39e23e1942c75effa from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:52.631690 1156 tablet_replica.cc:331] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.631825 1156 raft_consensus.cc:2241] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.631923 1156 raft_consensus.cc:2270] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.632303 1154 tablet_bootstrap.cc:492] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.633441 1154 tablet_bootstrap.cc:654] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.636279 877 tablet_service.cc:1515] Processing DeleteTablet for tablet c5d45854191247bf8172d0bbfd01b6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.636857 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c5d45854191247bf8172d0bbfd01b6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.637902 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ea00b4559e03410885180286b5e9fd10 (table default.Table_5d35d0525f924ed692497dbaf2287e8e [id=e7ad9b02d0f14bdfa3914844146a1e0b]) successfully deleted
I20250629 01:51:52.640704 1159 tablet_bootstrap.cc:492] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.640808 1159 ts_tablet_manager.cc:1397] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.002s
I20250629 01:51:52.641340 1159 raft_consensus.cc:357] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.641512 1159 raft_consensus.cc:383] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.641598 1159 raft_consensus.cc:738] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.641736 749 tablet_service.cc:1515] Processing DeleteTablet for tablet c5d45854191247bf8172d0bbfd01b6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.641873 1159 consensus_queue.cc:260] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:52.639461 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c5d45854191247bf8172d0bbfd01b6db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d45854191247bf8172d0bbfd01b6db already in progress: creating tablet
I20250629 01:51:52.642449 1159 ts_tablet_manager.cc:1428] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:52.642741 1159 tablet_bootstrap.cc:492] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.644176 1159 tablet_bootstrap.cc:654] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.645458 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cf4b52fae47f45ad9d77b0a97a854860
W20250629 01:51:52.645615 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 58403.5us
I20250629 01:51:52.645951 451 catalog_manager.cc:6652] ReplaceTablet: tablet cf4b52fae47f45ad9d77b0a97a854860 deleted and replaced by tablet 45461dc364d847d0b0272ac714559008
I20250629 01:51:52.646780 1154 tablet_bootstrap.cc:492] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.646785 2052 raft_consensus.cc:491] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.646942 1154 ts_tablet_manager.cc:1397] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20250629 01:51:52.646896 2052 raft_consensus.cc:513] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.647349 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet cf4b52fae47f45ad9d77b0a97a854860 with delete_type TABLET_DATA_DELETED (replaced by tablet 45461dc364d847d0b0272ac714559008) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.647361 2052 leader_election.cc:290] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.647547 1158 tablet_replica.cc:331] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.647615 1154 raft_consensus.cc:357] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.647696 1158 raft_consensus.cc:2241] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.647737 1154 raft_consensus.cc:383] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.648505 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cf4b52fae47f45ad9d77b0a97a854860 with delete_type TABLET_DATA_DELETED (replaced by tablet 45461dc364d847d0b0272ac714559008) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:52.648800 710 leader_election.cc:343] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:52.646775 877 tablet_service.cc:1515] Processing DeleteTablet for tablet cf4b52fae47f45ad9d77b0a97a854860 with delete_type TABLET_DATA_DELETED (replaced by tablet 45461dc364d847d0b0272ac714559008) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.648082 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a692f00ba244a80b9a6d07029f8d27e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.652537 712 leader_election.cc:343] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.647814 1158 raft_consensus.cc:2270] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.652624 712 leader_election.cc:304] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.653010 2049 raft_consensus.cc:2747] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.653131 1158 ts_tablet_manager.cc:1905] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.657706 2066 raft_consensus.cc:491] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.657801 2066 raft_consensus.cc:513] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.658192 2066 leader_election.cc:290] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.658416 1158 ts_tablet_manager.cc:1918] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:52.658538 1158 log.cc:1199] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/cf4b52fae47f45ad9d77b0a97a854860
I20250629 01:51:52.658710 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.658969 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:52.659080 988 leader_election.cc:343] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.659205 1158 ts_tablet_manager.cc:1939] T cf4b52fae47f45ad9d77b0a97a854860 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:52.659354 986 leader_election.cc:343] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.659448 986 leader_election.cc:304] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.652712 1154 raft_consensus.cc:738] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.659719 2066 raft_consensus.cc:2747] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.659739 1154 consensus_queue.cc:260] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.661213 1154 ts_tablet_manager.cc:1428] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250629 01:51:52.661252 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]) successfully deleted
I20250629 01:51:52.661433 1154 tablet_bootstrap.cc:492] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.664347 1154 tablet_bootstrap.cc:654] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.665246 1156 ts_tablet_manager.cc:1905] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.656152 1155 tablet_bootstrap.cc:654] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.668577 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 0323947b135a4308903b1eed1ca4db71 (table default.Table_ca2bcac172414e6c9f264cb2202a76fa [id=dfc541fca28746b8ba7e1f81a24c7fce]). Expected version 2 got 0
I20250629 01:51:52.668974 447 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_8b19cb4429164a77b7cd89742caf4afa [id=282409bfc86b422ba299a651ae80476f] alter complete (version 1)
W20250629 01:51:52.670764 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0323947b135a4308903b1eed1ca4db71 consensus state has no leader
I20250629 01:51:52.673965 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 35990d206ba64159ab05ab7b7b19d064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.674116 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 432680cae61946af9859760fe23110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.674414 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e962a020664b42ea918e5952dc43aade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.674881 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.675025 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 34172d05ae40417494cd8f67f4cb981e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.675180 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 47d4287317ec4788bd96ad174c31748d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.675354 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.675495 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f5941dd6381041538be6c92858d6ba13 with delete_type TABLET_DATA_DELETED (replaced by tablet 347fa6a12a744ee88058643b7143e39d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.677050 1158 tablet_replica.cc:331] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.677194 1158 raft_consensus.cc:2241] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.677304 1158 raft_consensus.cc:2270] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.677826 1158 ts_tablet_manager.cc:1905] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:52.680925 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1ea681bd91e54e4898bcfb72bed6198f because the tablet was not found. No further retry: Not found: Tablet not found: 1ea681bd91e54e4898bcfb72bed6198f
I20250629 01:51:52.682431 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 12b85f220f3947ca8110f2db937e7b68
I20250629 01:51:52.682936 448 catalog_manager.cc:6652] ReplaceTablet: tablet 12b85f220f3947ca8110f2db937e7b68 deleted and replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca
I20250629 01:51:52.683225 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 12b85f220f3947ca8110f2db937e7b68 with delete_type TABLET_DATA_DELETED (replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.683918 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 12b85f220f3947ca8110f2db937e7b68 with delete_type TABLET_DATA_DELETED (replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.683933 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 12b85f220f3947ca8110f2db937e7b68 with delete_type TABLET_DATA_DELETED (replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.684919 2078 tablet.cc:1722] T a2562570cb0846e19ebc97bdb43bd70d P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250629 01:51:52.688469 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:52.688663 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 8976cca816b442cc9a7cb4f23cc13016 (table default.Table_fadb5cdf8c8944018b9259b5fc865dec [id=3fb731afe0f54c88aa9a3643f46e4b1c]). Expected version 2 got 0
I20250629 01:51:52.688990 1159 tablet_bootstrap.cc:492] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.689090 1159 ts_tablet_manager.cc:1397] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.002s
I20250629 01:51:52.690012 1159 raft_consensus.cc:357] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.690141 1159 raft_consensus.cc:383] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.690186 1159 raft_consensus.cc:738] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.690292 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 351f4a7ae7e244999dda81d282de505d with delete_type TABLET_DATA_DELETED (replaced by tablet 897a4cefefe04f06b47070ec7fa1a33b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.690357 1159 consensus_queue.cc:260] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.690800 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 28eaa55b3b564e2e9cf698fda98a4c8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.690829 1159 ts_tablet_manager.cc:1428] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:52.691023 1159 tablet_bootstrap.cc:492] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.691164 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3a7fe7d34c6e4d919869404da9508244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.691617 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a7fe7d34c6e4d919869404da9508244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.692133 1159 tablet_bootstrap.cc:654] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.693354 1154 tablet_bootstrap.cc:492] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.693513 1154 ts_tablet_manager.cc:1397] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250629 01:51:52.693470 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7a16d1646321435cb77d2338d46598f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.694201 1154 raft_consensus.cc:357] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.694341 1154 raft_consensus.cc:383] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.694384 1154 raft_consensus.cc:738] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:52.694389 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a16d1646321435cb77d2338d46598f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.694569 1154 consensus_queue.cc:260] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.694944 877 tablet_service.cc:1515] Processing DeleteTablet for tablet e1fe3880e51345269e852014d0d60bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.693440 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b20baf3328ec42b1baa2585339b2e173"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.695416 2049 raft_consensus.cc:491] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.695459 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 49333302a27f4ac3b76a1e492b3c1b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:52.695540 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b20baf3328ec42b1baa2585339b2e173 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.695564 2049 raft_consensus.cc:513] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.695925 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 9624117e97974a47ac2bfe525da6f405 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.696056 2049 leader_election.cc:290] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.696789 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_581eff9507d54eb092e4ad71badc3c08" } new_table_name: "default.Table_441db41f85564f6ba3c4756f8d4f65a3" modify_external_catalogs: true
I20250629 01:51:52.697100 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "75ce1bd4af434f7e99092aafc0ab0118" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.697508 712 leader_election.cc:343] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:52.698752 710 leader_election.cc:343] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 75ce1bd4af434f7e99092aafc0ab0118
I20250629 01:51:52.698879 710 leader_election.cc:304] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.699259 2049 raft_consensus.cc:2747] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.701509 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 0b16b6840c24484b90218fffc1da9621 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.701542 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:52.704795 1156 ts_tablet_manager.cc:1918] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.704967 1156 log.cc:1199] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:52.705327 1156 ts_tablet_manager.cc:1939] T 9cf553b291854a6f9e7ed78b5db5f18c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.696067 1154 ts_tablet_manager.cc:1428] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:52.706079 1049 raft_consensus.cc:1273] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:52.706619 2049 consensus_queue.cc:1035] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:52.707656 1154 tablet_bootstrap.cc:492] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.708920 1154 tablet_bootstrap.cc:654] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.710131 1159 tablet_bootstrap.cc:492] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.710232 1159 ts_tablet_manager.cc:1397] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250629 01:51:52.710917 1159 raft_consensus.cc:357] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.711223 2066 raft_consensus.cc:491] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.711344 2066 raft_consensus.cc:513] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.711824 2066 leader_election.cc:290] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.712319 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "013ab393a0f14bdc89389349bcc5d24b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.712699 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "013ab393a0f14bdc89389349bcc5d24b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:52.712867 775 raft_consensus.cc:2466] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:52.713187 986 leader_election.cc:304] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:51:52.713311 988 leader_election.cc:343] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.713464 2066 raft_consensus.cc:2802] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:52.713519 2066 raft_consensus.cc:491] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:52.713567 2066 raft_consensus.cc:3058] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.711087 1159 raft_consensus.cc:383] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.713881 1159 raft_consensus.cc:738] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:51:52.713887 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3874404c3444d47a71f1dce7fb86ff8 consensus state has no leader
I20250629 01:51:52.714053 1159 consensus_queue.cc:260] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.714499 1159 ts_tablet_manager.cc:1428] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:52.714690 1159 tablet_bootstrap.cc:492] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.715708 2066 raft_consensus.cc:513] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.716346 2066 leader_election.cc:290] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.715843 1159 tablet_bootstrap.cc:654] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.719777 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 543242e4a5064268ae1e5519b569f682 with delete_type TABLET_DATA_DELETED (replaced by tablet df0fe7f5469d44fc8f6ac5c72377b48a) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.720463 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5d3980aa7f2c477889fb562e238587c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.720887 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d3980aa7f2c477889fb562e238587c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.720964 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d4dacbd3dbf34f52bcb81af671ddcf6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.721403 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4dacbd3dbf34f52bcb81af671ddcf6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.722517 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e3274fb78ed8400d9bc8be767b2e174c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.722903 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3274fb78ed8400d9bc8be767b2e174c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.723141 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9396b8200cc74c71b35de340567589a4" } new_table_name: "default.Table_1fbaed01d5a0409b94dc53e0e4a7cdc0" modify_external_catalogs: true
I20250629 01:51:52.723950 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_681f0348d61c4df39b9e7a9806daeaca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.724388 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_681f0348d61c4df39b9e7a9806daeaca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.726586 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4cc7e839241e447db039b261421a7fef
I20250629 01:51:52.726595 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "013ab393a0f14bdc89389349bcc5d24b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:52.726788 775 raft_consensus.cc:3058] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.727195 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cc7e839241e447db039b261421a7fef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.728226 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cc7e839241e447db039b261421a7fef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.716699 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "013ab393a0f14bdc89389349bcc5d24b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:52.732926 988 leader_election.cc:343] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.733454 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9cf553b291854a6f9e7ed78b5db5f18c (table default.Table_770438a7f7324802ab8dea0697fc2423 [id=d3d515038aa84669a3a64824de88f06e]) successfully deleted
I20250629 01:51:52.716974 2119 tablet.cc:1722] T a2562570cb0846e19ebc97bdb43bd70d P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250629 01:51:52.734997 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 4cc7e839241e447db039b261421a7fef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.735585 451 master_service.cc:948] ReplaceTablet: received request to replace tablet ec3937679b37476f934aceb30ac62d68 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:52.735718 2095 tablet.cc:1722] T a2562570cb0846e19ebc97bdb43bd70d P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 7
I20250629 01:51:52.736480 1861 raft_consensus.cc:491] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.736603 775 raft_consensus.cc:2466] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:52.736963 986 leader_election.cc:304] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:52.737396 2066 raft_consensus.cc:2802] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:52.737715 1154 tablet_bootstrap.cc:492] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.737808 1154 ts_tablet_manager.cc:1397] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20250629 01:51:52.740787 1158 ts_tablet_manager.cc:1918] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.741029 1158 log.cc:1199] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f5941dd6381041538be6c92858d6ba13
W20250629 01:51:52.741328 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4416bbec7d1c40df9a835b503d368d4c consensus state has no leader
I20250629 01:51:52.741369 1158 ts_tablet_manager.cc:1939] T f5941dd6381041538be6c92858d6ba13 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.741886 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f5941dd6381041538be6c92858d6ba13 (table default.Table_475fe7b17a5c4aac90faf6432e488d87 [id=0fb1390e9626454eba8e8d5c634e3943]) successfully deleted
W20250629 01:51:52.742416 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 12b85f220f3947ca8110f2db937e7b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12b85f220f3947ca8110f2db937e7b68 already in progress: creating tablet
W20250629 01:51:52.742954 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4cc7e839241e447db039b261421a7fef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cc7e839241e447db039b261421a7fef already in progress: creating tablet
I20250629 01:51:52.748302 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 59cc1690878f40e39e23e1942c75effa
I20250629 01:51:52.748747 455 catalog_manager.cc:6652] ReplaceTablet: tablet 59cc1690878f40e39e23e1942c75effa deleted and replaced by tablet 78e379d748434664b4e2684d3f6012ab
I20250629 01:51:52.748947 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 59cc1690878f40e39e23e1942c75effa with delete_type TABLET_DATA_DELETED (replaced by tablet 78e379d748434664b4e2684d3f6012ab) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.749956 2066 raft_consensus.cc:695] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:52.750209 2066 consensus_queue.cc:237] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.751407 1154 raft_consensus.cc:357] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.751559 1154 raft_consensus.cc:383] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.751607 1154 raft_consensus.cc:738] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.751762 1154 consensus_queue.cc:260] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.752169 1154 ts_tablet_manager.cc:1428] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250629 01:51:52.752384 1154 tablet_bootstrap.cc:492] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.753726 1154 tablet_bootstrap.cc:654] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.755596 1159 tablet_bootstrap.cc:492] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.756140 1156 tablet_replica.cc:331] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.756314 1156 raft_consensus.cc:2241] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.756302 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]). Expected version 2 got 0
I20250629 01:51:52.756573 1156 raft_consensus.cc:2270] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.757004 1156 ts_tablet_manager.cc:1905] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.757680 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
I20250629 01:51:52.757865 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:52.758009 449 catalog_manager.cc:5582] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:52.758466 449 catalog_manager.cc:5582] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:52.755862 1861 raft_consensus.cc:513] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.763931 738 tablet_service.cc:1468] Processing CreateTablet for tablet 3a21ee74cdad4be885a3018b31bdf2f3 (DEFAULT_TABLE table=default.Table_c02b87b2cd724dc9a010ea32cbbb057c [id=8dfd8d6ca67e4e3490c8793d955df977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.764295 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a21ee74cdad4be885a3018b31bdf2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.756009 1915 raft_consensus.cc:491] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.764413 1915 raft_consensus.cc:513] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.764873 1915 leader_election.cc:290] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.760447 877 tablet_service.cc:1468] Processing CreateTablet for tablet 3a21ee74cdad4be885a3018b31bdf2f3 (DEFAULT_TABLE table=default.Table_c02b87b2cd724dc9a010ea32cbbb057c [id=8dfd8d6ca67e4e3490c8793d955df977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.761492 895 tablet_service.cc:1468] Processing CreateTablet for tablet 77d497c0dc4f49caa22758d3d7517a95 (DEFAULT_TABLE table=default.Table_0d73122281254c95b722afbfdd68d670 [id=8ed3c90cf7494d2ea0f5c7ab2eaefb69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.762113 883 tablet_service.cc:1468] Processing CreateTablet for tablet f7841a49a73a40d6b72c2fa4709d8598 (DEFAULT_TABLE table=default.Table_6708008981374868897e9b1e1d6dba18 [id=09b8c290f906415c88b7997b3ec56db9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:52.765556 850 leader_election.cc:343] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:52.765761 1159 ts_tablet_manager.cc:1397] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.001s
I20250629 01:51:52.766323 1159 raft_consensus.cc:357] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.766456 1159 raft_consensus.cc:383] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.766494 1159 raft_consensus.cc:738] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.767338 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 59cc1690878f40e39e23e1942c75effa with delete_type TABLET_DATA_DELETED (replaced by tablet 78e379d748434664b4e2684d3f6012ab) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.762600 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e83009c1599a43d18d1c73c0101ab0be" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.763077 879 tablet_service.cc:1468] Processing CreateTablet for tablet 45c5a48bec6849e0994562c561dfee5a (DEFAULT_TABLE table=default.Table_3a542c87991a41379b067ca98b5235fa [id=de89f6bac102403cab9b0d7ba53271cb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:52.767575 850 leader_election.cc:343] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:52.767647 850 leader_election.cc:304] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.767920 1915 raft_consensus.cc:2747] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.768957 1159 consensus_queue.cc:260] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.770792 755 tablet_service.cc:1468] Processing CreateTablet for tablet 77d497c0dc4f49caa22758d3d7517a95 (DEFAULT_TABLE table=default.Table_0d73122281254c95b722afbfdd68d670 [id=8ed3c90cf7494d2ea0f5c7ab2eaefb69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.769366 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7841a49a73a40d6b72c2fa4709d8598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.771401 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c5a48bec6849e0994562c561dfee5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.771443 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d497c0dc4f49caa22758d3d7517a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.773231 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d497c0dc4f49caa22758d3d7517a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.773622 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a21ee74cdad4be885a3018b31bdf2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.776238 745 tablet_service.cc:1468] Processing CreateTablet for tablet f7841a49a73a40d6b72c2fa4709d8598 (DEFAULT_TABLE table=default.Table_6708008981374868897e9b1e1d6dba18 [id=09b8c290f906415c88b7997b3ec56db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.776456 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14ca2bdaa17c48b59043ada398aa4bc4" } modify_external_catalogs: true
I20250629 01:51:52.776517 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7841a49a73a40d6b72c2fa4709d8598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.776585 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14ca2bdaa17c48b59043ada398aa4bc4" } modify_external_catalogs: true
I20250629 01:51:52.776831 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f7841a49a73a40d6b72c2fa4709d8598 (DEFAULT_TABLE table=default.Table_6708008981374868897e9b1e1d6dba18 [id=09b8c290f906415c88b7997b3ec56db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.777097 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7841a49a73a40d6b72c2fa4709d8598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.778874 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 3a21ee74cdad4be885a3018b31bdf2f3 (DEFAULT_TABLE table=default.Table_c02b87b2cd724dc9a010ea32cbbb057c [id=8dfd8d6ca67e4e3490c8793d955df977]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.779120 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a21ee74cdad4be885a3018b31bdf2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.780066 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 59cc1690878f40e39e23e1942c75effa with delete_type TABLET_DATA_DELETED (replaced by tablet 78e379d748434664b4e2684d3f6012ab) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:52.780117 712 leader_election.cc:343] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.780333 1861 leader_election.cc:290] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.780336 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 77d497c0dc4f49caa22758d3d7517a95 (DEFAULT_TABLE table=default.Table_0d73122281254c95b722afbfdd68d670 [id=8ed3c90cf7494d2ea0f5c7ab2eaefb69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.780591 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d497c0dc4f49caa22758d3d7517a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.781612 1915 raft_consensus.cc:491] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.781718 1915 raft_consensus.cc:513] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.782114 1915 leader_election.cc:290] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.783231 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:52.783246 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8976cca816b442cc9a7cb4f23cc13016" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:52.783728 850 leader_election.cc:304] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.783998 1915 raft_consensus.cc:2747] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:52.784304 710 leader_election.cc:343] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:52.784421 710 leader_election.cc:304] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.784624 1861 raft_consensus.cc:2747] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:52.784904 710 consensus_peers.cc:489] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf4b52fae47f45ad9d77b0a97a854860. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:52.785640 1159 ts_tablet_manager.cc:1428] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250629 01:51:52.785650 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12] alter complete (version 3)
I20250629 01:51:52.785862 1159 tablet_bootstrap.cc:492] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.793000 1156 ts_tablet_manager.cc:1918] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.793148 1156 log.cc:1199] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:52.792843 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:52.793464 1156 ts_tablet_manager.cc:1939] T 81b32bac5a2144d1943285f5f446d53d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.793669 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.793623 450 catalog_manager.cc:5582] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:52.793852 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet ef976703d61442f8987ff32070a33592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.793949 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:52.794030 1158 tablet_replica.cc:331] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.794183 1158 raft_consensus.cc:2241] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.794301 1158 raft_consensus.cc:2270] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.794312 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.794431 1159 tablet_bootstrap.cc:654] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.794757 1156 tablet_replica.cc:331] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:52.798163 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 59cc1690878f40e39e23e1942c75effa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59cc1690878f40e39e23e1942c75effa already in progress: creating tablet
I20250629 01:51:52.798434 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 81b32bac5a2144d1943285f5f446d53d (table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]) successfully deleted
I20250629 01:51:52.802282 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92526d1aecd94064817e09aca50dd8d0" } modify_external_catalogs: true
I20250629 01:51:52.802415 1158 ts_tablet_manager.cc:1905] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.802450 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92526d1aecd94064817e09aca50dd8d0" } modify_external_catalogs: true
I20250629 01:51:52.803026 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b92827e379f148d4bd76b3e578adc695" } new_table_name: "default.Table_97c5093f3a1542f5bb7fc463c1955f9e" modify_external_catalogs: true
I20250629 01:51:52.803181 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_de40e2ec822b4c29a10c638c0a7d6f46" } new_table_name: "default.Table_0814dc6691b040e1a9f69f258973cf28" modify_external_catalogs: true
I20250629 01:51:52.804025 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 730178bc660f4af596007721df96198f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:52.804221 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3b6a5b57d8964e5085b449b331a8702e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.804884 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b6a5b57d8964e5085b449b331a8702e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.805903 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bd5150eff1ad43588a9e70dab47744e3 with delete_type TABLET_DATA_DELETED (replaced by tablet 0bef2747f0c04f498079906e40bac24c) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.808141 1157 ts_tablet_manager.cc:1918] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20250629 01:51:52.808225 1157 log.cc:1199] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/57f7f71ab5e54d249141c89d9e806111
I20250629 01:51:52.810465 1157 ts_tablet_manager.cc:1939] T 57f7f71ab5e54d249141c89d9e806111 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:52.811590 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 62369ffc1d4a4628bb24254877d6dee6
I20250629 01:51:52.812314 1156 raft_consensus.cc:2241] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.812309 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 62369ffc1d4a4628bb24254877d6dee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.812475 1156 raft_consensus.cc:2270] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:52.812599 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e304f0bfe13846f29e6390d9904a4f57 consensus state has no leader
I20250629 01:51:52.812940 1156 ts_tablet_manager.cc:1905] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:52.813757 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90eef69522e54b12b849d94e56355921 consensus state has no leader
I20250629 01:51:52.813824 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ec3937679b37476f934aceb30ac62d68
I20250629 01:51:52.814271 451 catalog_manager.cc:6652] ReplaceTablet: tablet ec3937679b37476f934aceb30ac62d68 deleted and replaced by tablet 966e379adef8435887a2d8f39bb99968
I20250629 01:51:52.814389 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet ec3937679b37476f934aceb30ac62d68 with delete_type TABLET_DATA_DELETED (replaced by tablet 966e379adef8435887a2d8f39bb99968) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.794826 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e83009c1599a43d18d1c73c0101ab0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:52.814392 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 63986.9us
I20250629 01:51:52.811968 1159 tablet_bootstrap.cc:492] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.814591 1159 ts_tablet_manager.cc:1397] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.001s
I20250629 01:51:52.815169 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ec3937679b37476f934aceb30ac62d68 with delete_type TABLET_DATA_DELETED (replaced by tablet 966e379adef8435887a2d8f39bb99968) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.816610 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:52.796964 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.797369 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.815160 1159 raft_consensus.cc:357] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.817490 1159 raft_consensus.cc:383] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:51:52.797598 2111 log.cc:927] Time spent T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250629 01:51:52.817551 1159 raft_consensus.cc:738] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.798002 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.812070 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 62369ffc1d4a4628bb24254877d6dee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.817701 1159 consensus_queue.cc:260] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.818145 1159 ts_tablet_manager.cc:1428] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:52.818415 1158 ts_tablet_manager.cc:1918] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.818509 1158 log.cc:1199] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ef976703d61442f8987ff32070a33592
I20250629 01:51:52.818751 1158 ts_tablet_manager.cc:1939] T ef976703d61442f8987ff32070a33592 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.818809 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 62369ffc1d4a4628bb24254877d6dee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.819265 2078 tablet.cc:1722] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:52.820292 1159 tablet_bootstrap.cc:492] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.820474 2095 tablet.cc:1722] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:52.821534 1159 tablet_bootstrap.cc:654] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.822391 2078 tablet.cc:1722] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:52.824220 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:52.825764 755 tablet_service.cc:1468] Processing CreateTablet for tablet 45c5a48bec6849e0994562c561dfee5a (DEFAULT_TABLE table=default.Table_3a542c87991a41379b067ca98b5235fa [id=de89f6bac102403cab9b0d7ba53271cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.826095 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c5a48bec6849e0994562c561dfee5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.826572 879 tablet_service.cc:1515] Processing DeleteTablet for tablet ec3937679b37476f934aceb30ac62d68 with delete_type TABLET_DATA_DELETED (replaced by tablet 966e379adef8435887a2d8f39bb99968) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.827236 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 45c5a48bec6849e0994562c561dfee5a (DEFAULT_TABLE table=default.Table_3a542c87991a41379b067ca98b5235fa [id=de89f6bac102403cab9b0d7ba53271cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.827577 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45c5a48bec6849e0994562c561dfee5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.829830 2095 tablet.cc:1722] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:52.830770 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d4a2315a9cc24966b7cd234220e6c247" } modify_external_catalogs: true
I20250629 01:51:52.830941 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d4a2315a9cc24966b7cd234220e6c247" } modify_external_catalogs: true
I20250629 01:51:52.834093 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7054a2ae6968462bbf1a604dc08243fc" } modify_external_catalogs: true
I20250629 01:51:52.834331 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7054a2ae6968462bbf1a604dc08243fc" } modify_external_catalogs: true
I20250629 01:51:52.836277 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cd7226b7561e4b56a60721679781b7a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.836797 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd7226b7561e4b56a60721679781b7a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.838725 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_56965d558fb9442fa7c189c772e55937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.839133 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56965d558fb9442fa7c189c772e55937 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.840018 1156 ts_tablet_manager.cc:1918] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.840157 1156 log.cc:1199] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:52.840220 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:52.840989 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ef976703d61442f8987ff32070a33592 (table default.Table_f220872b43b3433694c2f522f6e54acd [id=b1662df1e42f43eeae59f02a3e446006]) successfully deleted
W20250629 01:51:52.841727 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 62369ffc1d4a4628bb24254877d6dee6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62369ffc1d4a4628bb24254877d6dee6 already in progress: creating tablet
W20250629 01:51:52.842383 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ec3937679b37476f934aceb30ac62d68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec3937679b37476f934aceb30ac62d68 already in progress: creating tablet
I20250629 01:51:52.852572 2066 raft_consensus.cc:491] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.852676 2066 raft_consensus.cc:513] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.853149 2066 leader_election.cc:290] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:52.854362 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "add482b533d045e7a3010064fae730cf" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.854717 988 leader_election.cc:343] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.856900 1915 raft_consensus.cc:491] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.857005 1915 raft_consensus.cc:513] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.857456 1915 leader_election.cc:290] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:52.857765 879 tablet_service.cc:1468] Processing CreateTablet for tablet ec0365d5ad654a7a9df622a27bfbe6fd (DEFAULT_TABLE table=default.Table_985b5a9e7b454754a12e4e3fdb38a3a4 [id=0c6d1537cb3d4f0a933d8d2f9a46da65]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:52.858004 850 leader_election.cc:343] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 351f4a7ae7e244999dda81d282de505d
I20250629 01:51:52.858062 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0365d5ad654a7a9df622a27bfbe6fd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:52.858130 850 leader_election.cc:343] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 351f4a7ae7e244999dda81d282de505d
I20250629 01:51:52.858188 850 leader_election.cc:304] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.858613 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 57f7f71ab5e54d249141c89d9e806111 (table default.Table_ab9e746444a24266aca5439c51b30a27 [id=60094a0a7c134476a2a4e44d8555c5d6]) successfully deleted
W20250629 01:51:52.858805 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8975c1deaaf44761be791f460b53ec74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8975c1deaaf44761be791f460b53ec74 already in progress: creating tablet
W20250629 01:51:52.859416 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet faca0e904a04456d9aead23abd44e00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faca0e904a04456d9aead23abd44e00a already in progress: creating tablet
I20250629 01:51:52.859583 1915 raft_consensus.cc:2747] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.860246 895 tablet_service.cc:1468] Processing CreateTablet for tablet caa63fe10ff9487a9fc16479285c5abd (DEFAULT_TABLE table=default.Table_17373cbe989b4dfcb4c39ed0f708d216 [id=40a8686a877a453a8bbd6906c3cac419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.860518 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa63fe10ff9487a9fc16479285c5abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.860934 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ec0365d5ad654a7a9df622a27bfbe6fd (DEFAULT_TABLE table=default.Table_985b5a9e7b454754a12e4e3fdb38a3a4 [id=0c6d1537cb3d4f0a933d8d2f9a46da65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.861193 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0365d5ad654a7a9df622a27bfbe6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.861181 883 tablet_service.cc:1468] Processing CreateTablet for tablet 64ed16be3e684e32b6eb80a6a23c86ac (DEFAULT_TABLE table=default.Table_9ffe7a3207f44327bb5b618169fa2aaf [id=99f735943f664abd9474e9db8c90c579]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.861442 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ed16be3e684e32b6eb80a6a23c86ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.863381 1031 tablet_service.cc:1468] Processing CreateTablet for tablet caa63fe10ff9487a9fc16479285c5abd (DEFAULT_TABLE table=default.Table_17373cbe989b4dfcb4c39ed0f708d216 [id=40a8686a877a453a8bbd6906c3cac419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.863654 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa63fe10ff9487a9fc16479285c5abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.868134 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 64ed16be3e684e32b6eb80a6a23c86ac (DEFAULT_TABLE table=default.Table_9ffe7a3207f44327bb5b618169fa2aaf [id=99f735943f664abd9474e9db8c90c579]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.868422 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ed16be3e684e32b6eb80a6a23c86ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.869774 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 730178bc660f4af596007721df96198f
I20250629 01:51:52.870113 452 catalog_manager.cc:6652] ReplaceTablet: tablet 730178bc660f4af596007721df96198f deleted and replaced by tablet 9ea65305417e45fdb1a47768cc1ee054
I20250629 01:51:52.872313 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d7032fa31e084ecc8861910661af6259" } new_table_name: "default.Table_c456f245c84847adb29db0e98c23bde7" modify_external_catalogs: true
I20250629 01:51:52.849038 2049 raft_consensus.cc:491] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.873358 2049 raft_consensus.cc:513] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.873651 1154 tablet_bootstrap.cc:492] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.873747 1154 ts_tablet_manager.cc:1397] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.000s
I20250629 01:51:52.873885 2049 leader_election.cc:290] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.873972 916 raft_consensus.cc:3058] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:52.874296 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a485e0e6e2f04aa6bd67a62f6d30be26" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:52.874354 1154 raft_consensus.cc:357] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.874481 1154 raft_consensus.cc:383] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.874533 1154 raft_consensus.cc:738] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.854028 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "add482b533d045e7a3010064fae730cf" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:52.874680 1154 consensus_queue.cc:260] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:52.874958 986 leader_election.cc:343] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.875036 986 leader_election.cc:304] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.875068 1154 ts_tablet_manager.cc:1428] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:52.875293 1154 tablet_bootstrap.cc:492] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.875365 2066 raft_consensus.cc:2747] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:52.875756 712 leader_election.cc:343] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.876335 1156 ts_tablet_manager.cc:1939] T df66984acfa44a99bf69096aff6d49ab P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:52.877914 1154 tablet_bootstrap.cc:654] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.864396 755 tablet_service.cc:1468] Processing CreateTablet for tablet ec0365d5ad654a7a9df622a27bfbe6fd (DEFAULT_TABLE table=default.Table_985b5a9e7b454754a12e4e3fdb38a3a4 [id=0c6d1537cb3d4f0a933d8d2f9a46da65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.882452 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec0365d5ad654a7a9df622a27bfbe6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.885205 877 tablet_service.cc:1468] Processing CreateTablet for tablet 29e05dd9908c4390a55b55e2eef8d4b2 (DEFAULT_TABLE table=default.Table_7ca2e46142c5424ab8f471c13d635ca2 [id=97190140b4a849aaa9758958f076c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.885466 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e05dd9908c4390a55b55e2eef8d4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.885952 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 730178bc660f4af596007721df96198f with delete_type TABLET_DATA_DELETED (replaced by tablet 9ea65305417e45fdb1a47768cc1ee054) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.887085 1154 tablet_bootstrap.cc:492] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.887177 1154 ts_tablet_manager.cc:1397] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20250629 01:51:52.867551 745 tablet_service.cc:1468] Processing CreateTablet for tablet caa63fe10ff9487a9fc16479285c5abd (DEFAULT_TABLE table=default.Table_17373cbe989b4dfcb4c39ed0f708d216 [id=40a8686a877a453a8bbd6906c3cac419]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.887506 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa63fe10ff9487a9fc16479285c5abd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.887596 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 730178bc660f4af596007721df96198f with delete_type TABLET_DATA_DELETED (replaced by tablet 9ea65305417e45fdb1a47768cc1ee054) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.887761 1154 raft_consensus.cc:357] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.887885 1154 raft_consensus.cc:383] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.887923 1154 raft_consensus.cc:738] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.888054 1154 consensus_queue.cc:260] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.888252 749 tablet_service.cc:1468] Processing CreateTablet for tablet 64ed16be3e684e32b6eb80a6a23c86ac (DEFAULT_TABLE table=default.Table_9ffe7a3207f44327bb5b618169fa2aaf [id=99f735943f664abd9474e9db8c90c579]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:52.877456 710 leader_election.cc:343] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:52.888564 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64ed16be3e684e32b6eb80a6a23c86ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.888600 710 leader_election.cc:304] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.889853 1154 ts_tablet_manager.cc:1428] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:52.889902 738 tablet_service.cc:1468] Processing CreateTablet for tablet 29e05dd9908c4390a55b55e2eef8d4b2 (DEFAULT_TABLE table=default.Table_7ca2e46142c5424ab8f471c13d635ca2 [id=97190140b4a849aaa9758958f076c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.890060 1154 tablet_bootstrap.cc:492] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.859145 1155 tablet_bootstrap.cc:492] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:52.890882 1155 ts_tablet_manager.cc:1397] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.265s user 0.005s sys 0.000s
I20250629 01:51:52.891609 1155 raft_consensus.cc:357] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.891757 1155 raft_consensus.cc:383] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.891829 1155 raft_consensus.cc:738] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.891997 1155 consensus_queue.cc:260] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.892428 1155 ts_tablet_manager.cc:1428] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:52.892547 1154 tablet_bootstrap.cc:654] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.892678 1155 tablet_bootstrap.cc:492] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:52.893590 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e05dd9908c4390a55b55e2eef8d4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.894145 1155 tablet_bootstrap.cc:654] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.895134 2049 raft_consensus.cc:2747] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.895401 2049 raft_consensus.cc:491] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.895526 2049 raft_consensus.cc:513] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.896099 2049 leader_election.cc:290] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.896329 2119 tablet.cc:1722] T 8976cca816b442cc9a7cb4f23cc13016 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:52.902146 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ac9d77381f7e4241aa546eaea3077168
I20250629 01:51:52.902264 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 64d3efa5b59c4ae6af52c7490c316d7e
I20250629 01:51:52.902707 447 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_600f51cec79c492496ab746496a0ede3 [id=9ff841a004c344c688f5abf6b59b5dce] alter complete (version 7)
I20250629 01:51:52.906093 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e841a0c33e504a5081bae39d731106cb
I20250629 01:51:52.909623 1154 tablet_bootstrap.cc:492] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.909716 1154 ts_tablet_manager.cc:1397] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250629 01:51:52.909776 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e841a0c33e504a5081bae39d731106cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.910153 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 64d3efa5b59c4ae6af52c7490c316d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.910266 1154 raft_consensus.cc:357] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.910459 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9d77381f7e4241aa546eaea3077168 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.910557 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 64d3efa5b59c4ae6af52c7490c316d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.910610 1154 raft_consensus.cc:383] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.910662 1154 raft_consensus.cc:738] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.910845 1154 consensus_queue.cc:260] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.910897 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 with delete_type TABLET_DATA_DELETED (replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.911209 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec with delete_type TABLET_DATA_DELETED (replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.911533 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e841a0c33e504a5081bae39d731106cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:52.912261 710 leader_election.cc:343] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e962a020664b42ea918e5952dc43aade
I20250629 01:51:52.916527 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 730178bc660f4af596007721df96198f with delete_type TABLET_DATA_DELETED (replaced by tablet 9ea65305417e45fdb1a47768cc1ee054) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:52.916529 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2c05f26e7f594408b830d3c545b58966"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.916767 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e841a0c33e504a5081bae39d731106cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.916945 895 tablet_service.cc:1515] Processing DeleteTablet for tablet e82b1352bb4d49769808dc402c85890f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:52.916991 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c05f26e7f594408b830d3c545b58966 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.917354 1154 ts_tablet_manager.cc:1428] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250629 01:51:52.917577 1154 tablet_bootstrap.cc:492] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.918322 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_91cf07f495b64649afe5b6d70efa7dba" } new_table_name: "default.Table_2b8e1cf9c4eb40c2ba24d1490398e140" modify_external_catalogs: true
I20250629 01:51:52.918629 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a1dcfdd16d0746d0a7ff69766d0b967d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.918761 1154 tablet_bootstrap.cc:654] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:52.919016 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1dcfdd16d0746d0a7ff69766d0b967d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.920120 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_09663e9d00cb47ecb9d43ed1ab066144"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:52.920361 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 64d3efa5b59c4ae6af52c7490c316d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:52.920542 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09663e9d00cb47ecb9d43ed1ab066144 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.920547 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e962a020664b42ea918e5952dc43aade" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.920935 712 leader_election.cc:343] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.921044 712 leader_election.cc:304] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:52.921389 1861 raft_consensus.cc:2747] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.921854 745 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9d77381f7e4241aa546eaea3077168 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:52.922370 1158 tablet_replica.cc:331] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:52.922514 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ac9d77381f7e4241aa546eaea3077168 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9d77381f7e4241aa546eaea3077168 already in progress: creating tablet
W20250629 01:51:52.922724 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 64d3efa5b59c4ae6af52c7490c316d7e because the tablet was not found. No further retry: Not found: Tablet not found: 64d3efa5b59c4ae6af52c7490c316d7e
I20250629 01:51:52.922844 1154 tablet_bootstrap.cc:492] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.922938 1154 ts_tablet_manager.cc:1397] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:52.922752 1158 raft_consensus.cc:2241] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.924263 1158 raft_consensus.cc:2270] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.924964 1158 ts_tablet_manager.cc:1905] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.923513 1154 raft_consensus.cc:357] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.925974 1154 raft_consensus.cc:383] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.926038 1154 raft_consensus.cc:738] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.926155 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet df66984acfa44a99bf69096aff6d49ab (table default.Table_abc4bc33793d4bee824f97f24a172593 [id=8a6a13c4f09547aa8ca713ab9892f557]) successfully deleted
I20250629 01:51:52.926376 1157 tablet_replica.cc:331] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:52.926478 1154 consensus_queue.cc:260] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.926549 1157 raft_consensus.cc:2241] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.926664 1157 raft_consensus.cc:2270] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.927096 1157 ts_tablet_manager.cc:1905] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.927227 879 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9d77381f7e4241aa546eaea3077168 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:52.927268 1156 tablet_replica.cc:331] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:52.930498 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4eb34771188b4b8d957400d4e3d8f3aa" } modify_external_catalogs: true
I20250629 01:51:52.930655 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4eb34771188b4b8d957400d4e3d8f3aa" } modify_external_catalogs: true
I20250629 01:51:52.932564 1154 ts_tablet_manager.cc:1428] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250629 01:51:52.932765 1154 tablet_bootstrap.cc:492] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.933173 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_583444e245fc499bb4f38e7921e59da3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.933575 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_583444e245fc499bb4f38e7921e59da3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.934052 1154 tablet_bootstrap.cc:654] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.936648 877 tablet_service.cc:1468] Processing CreateTablet for tablet abbefa9110cc41989c33d25e3ee8ea07 (DEFAULT_TABLE table=default.Table_abe281abbab94551a471e46d3908fc39 [id=98380c6e09a445aaa12e1b73acc4ec90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.937492 883 tablet_service.cc:1468] Processing CreateTablet for tablet 64d3efa5b59c4ae6af52c7490c316d7e (DEFAULT_TABLE table=default.Table_d4a2315a9cc24966b7cd234220e6c247 [id=11a8543fb0624f179dfa67a101cc3588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.938169 895 tablet_service.cc:1468] Processing CreateTablet for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 (DEFAULT_TABLE table=default.Table_aaa30d8b97a8406fb1f5fe4c5f309010 [id=b5a2e36c87aa456f949f71809b1a9801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.938372 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6d323c26638e4b2c9a01abd8feee7412"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:52.938784 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d323c26638e4b2c9a01abd8feee7412 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:52.939136 879 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1659dc2ed8482d81063ad64de072e5 (DEFAULT_TABLE table=default.Table_af06514069d14e6caea07fb498630608 [id=e5cae4d7f4ca4a4ca94cb3c79f7eb87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.940711 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1659dc2ed8482d81063ad64de072e5 (DEFAULT_TABLE table=default.Table_af06514069d14e6caea07fb498630608 [id=e5cae4d7f4ca4a4ca94cb3c79f7eb87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.940711 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 29e05dd9908c4390a55b55e2eef8d4b2 (DEFAULT_TABLE table=default.Table_7ca2e46142c5424ab8f471c13d635ca2 [id=97190140b4a849aaa9758958f076c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.940802 745 tablet_service.cc:1468] Processing CreateTablet for tablet 3f1659dc2ed8482d81063ad64de072e5 (DEFAULT_TABLE table=default.Table_af06514069d14e6caea07fb498630608 [id=e5cae4d7f4ca4a4ca94cb3c79f7eb87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.941040 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1659dc2ed8482d81063ad64de072e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.941219 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29e05dd9908c4390a55b55e2eef8d4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.941243 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1659dc2ed8482d81063ad64de072e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.942202 1154 tablet_bootstrap.cc:492] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.942294 1154 ts_tablet_manager.cc:1397] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20250629 01:51:52.942821 1154 raft_consensus.cc:357] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.942924 1154 raft_consensus.cc:383] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.942962 1154 raft_consensus.cc:738] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.943099 1154 consensus_queue.cc:260] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.944325 1156 raft_consensus.cc:2241] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.944512 1156 raft_consensus.cc:2270] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.944875 1154 ts_tablet_manager.cc:1428] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:52.944972 1156 ts_tablet_manager.cc:1905] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.945097 1154 tablet_bootstrap.cc:492] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:52.946316 1154 tablet_bootstrap.cc:654] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:52.948758 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 90eef69522e54b12b849d94e56355921 (table default.Table_97c5093f3a1542f5bb7fc463c1955f9e [id=6faf9bc8cce84174a48928674dc25702], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 90eef69522e54b12b849d94e56355921 consensus state has no leader
I20250629 01:51:52.949436 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 (DEFAULT_TABLE table=default.Table_aaa30d8b97a8406fb1f5fe4c5f309010 [id=b5a2e36c87aa456f949f71809b1a9801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.949843 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ca3db3bedd4ce8bd8159512a15a3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.953372 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d3efa5b59c4ae6af52c7490c316d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.953516 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abbefa9110cc41989c33d25e3ee8ea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.954003 749 tablet_service.cc:1468] Processing CreateTablet for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 (DEFAULT_TABLE table=default.Table_aaa30d8b97a8406fb1f5fe4c5f309010 [id=b5a2e36c87aa456f949f71809b1a9801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.954339 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ca3db3bedd4ce8bd8159512a15a3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.957793 738 tablet_service.cc:1468] Processing CreateTablet for tablet abbefa9110cc41989c33d25e3ee8ea07 (DEFAULT_TABLE table=default.Table_abe281abbab94551a471e46d3908fc39 [id=98380c6e09a445aaa12e1b73acc4ec90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.958525 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ca3db3bedd4ce8bd8159512a15a3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.958592 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abbefa9110cc41989c33d25e3ee8ea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.960721 1159 tablet_bootstrap.cc:492] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.960831 1159 ts_tablet_manager.cc:1397] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.019s
I20250629 01:51:52.961412 1159 raft_consensus.cc:357] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.961550 1159 raft_consensus.cc:383] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.961601 1159 raft_consensus.cc:738] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.961805 1159 consensus_queue.cc:260] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.963394 1159 ts_tablet_manager.cc:1428] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:52.963608 1159 tablet_bootstrap.cc:492] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:52.964912 1159 tablet_bootstrap.cc:654] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:52.971590 2126 raft_consensus.cc:491] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.971715 2126 raft_consensus.cc:513] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:52.972298 2126 leader_election.cc:290] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:52.972815 755 tablet_service.cc:1468] Processing CreateTablet for tablet 64d3efa5b59c4ae6af52c7490c316d7e (DEFAULT_TABLE table=default.Table_d4a2315a9cc24966b7cd234220e6c247 [id=11a8543fb0624f179dfa67a101cc3588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.973134 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d3efa5b59c4ae6af52c7490c316d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.973403 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56b9ef027c9d42048e4a6dff221df793" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:52.973802 986 leader_election.cc:343] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.974603 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56b9ef027c9d42048e4a6dff221df793" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:52.975008 988 leader_election.cc:343] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:52.975114 988 leader_election.cc:304] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:52.975483 2126 raft_consensus.cc:2747] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:52.976311 889 tablet_service.cc:1468] Processing CreateTablet for tablet 4de6fca85fb643b998be42e9bc7bdbb6 (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.976598 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de6fca85fb643b998be42e9bc7bdbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.976652 750 tablet_service.cc:1468] Processing CreateTablet for tablet db191090410941e68cf3221b9cc62f51 (DEFAULT_TABLE table=default.Table_7039f5dc37154a69870a9955aa07d568 [id=b91b10831b3b4e7893edc082174550bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.976949 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db191090410941e68cf3221b9cc62f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.976970 1158 ts_tablet_manager.cc:1918] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:52.977057 1158 log.cc:1199] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:52.977283 1158 ts_tablet_manager.cc:1939] T f8b38c20eb1849b1a9f8acd40315eef5 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:52.977587 1158 tablet_replica.cc:331] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:52.977722 1158 raft_consensus.cc:2241] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:52.977829 1158 raft_consensus.cc:2270] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:52.980306 878 tablet_service.cc:1468] Processing CreateTablet for tablet cc65674750c64114b999e0e292a851f9 (DEFAULT_TABLE table=default.Table_f52b1033085542258b846453f2b91f3d [id=f05487296c954ad08343c2a49b1934ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.980581 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc65674750c64114b999e0e292a851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.982463 750 tablet_service.cc:1468] Processing CreateTablet for tablet 4de6fca85fb643b998be42e9bc7bdbb6 (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.982725 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de6fca85fb643b998be42e9bc7bdbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.983747 743 tablet_service.cc:1468] Processing CreateTablet for tablet cc65674750c64114b999e0e292a851f9 (DEFAULT_TABLE table=default.Table_f52b1033085542258b846453f2b91f3d [id=f05487296c954ad08343c2a49b1934ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.984046 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc65674750c64114b999e0e292a851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.984066 740 tablet_service.cc:1468] Processing CreateTablet for tablet dc1b24606e6f4decb30364fa4fafc147 (DEFAULT_TABLE table=default.Table_8a1e4394a537495b9bf3f9e4335f7936 [id=e6a1c068fbca47308ae288b6d28fc63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.984366 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1b24606e6f4decb30364fa4fafc147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.988621 877 tablet_service.cc:1468] Processing CreateTablet for tablet db191090410941e68cf3221b9cc62f51 (DEFAULT_TABLE table=default.Table_7039f5dc37154a69870a9955aa07d568 [id=b91b10831b3b4e7893edc082174550bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.988716 1159 tablet_bootstrap.cc:492] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:52.988833 1159 ts_tablet_manager.cc:1397] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250629 01:51:52.988873 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db191090410941e68cf3221b9cc62f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.989537 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 64d3efa5b59c4ae6af52c7490c316d7e (DEFAULT_TABLE table=default.Table_d4a2315a9cc24966b7cd234220e6c247 [id=11a8543fb0624f179dfa67a101cc3588]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.989887 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d3efa5b59c4ae6af52c7490c316d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.990054 1158 ts_tablet_manager.cc:1905] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:52.991531 890 tablet_service.cc:1468] Processing CreateTablet for tablet dc1b24606e6f4decb30364fa4fafc147 (DEFAULT_TABLE table=default.Table_8a1e4394a537495b9bf3f9e4335f7936 [id=e6a1c068fbca47308ae288b6d28fc63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:52.991787 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1b24606e6f4decb30364fa4fafc147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:52.991945 1159 raft_consensus.cc:357] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.992058 1159 raft_consensus.cc:383] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.992106 1159 raft_consensus.cc:738] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:52.992249 1159 consensus_queue.cc:260] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.994334 1154 tablet_bootstrap.cc:492] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:52.994467 1154 ts_tablet_manager.cc:1397] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.003s
I20250629 01:51:52.995018 1154 raft_consensus.cc:357] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.990355 1915 raft_consensus.cc:491] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.996184 1915 raft_consensus.cc:513] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:52.996614 1915 leader_election.cc:290] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:52.997107 1154 raft_consensus.cc:383] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:52.997186 1154 raft_consensus.cc:738] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:52.998298 850 leader_election.cc:343] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: bd5150eff1ad43588a9e70dab47744e3
I20250629 01:51:52.998459 2052 raft_consensus.cc:491] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:52.998618 2052 raft_consensus.cc:513] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:52.999861 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "683c87ea2ad44cecb2366d4a846d9195" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.000070 850 leader_election.cc:343] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: bd5150eff1ad43588a9e70dab47744e3
I20250629 01:51:53.000209 850 leader_election.cc:304] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.000495 1915 raft_consensus.cc:2747] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.000734 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f8b38c20eb1849b1a9f8acd40315eef5 (table default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]) successfully deleted
I20250629 01:51:53.000878 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1659dc2ed8482d81063ad64de072e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.008282 1155 tablet_bootstrap.cc:492] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.005918 1154 consensus_queue.cc:260] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.009685 1154 ts_tablet_manager.cc:1428] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250629 01:51:53.009930 1154 tablet_bootstrap.cc:492] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.011173 1154 tablet_bootstrap.cc:654] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.014882 1157 ts_tablet_manager.cc:1918] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:53.015496 712 leader_election.cc:343] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.016953 1157 log.cc:1199] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a90b206d58d04acdb1a8cb0c45d68c31
I20250629 01:51:53.021000 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 4de6fca85fb643b998be42e9bc7bdbb6 (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.021286 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de6fca85fb643b998be42e9bc7bdbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.026891 1157 ts_tablet_manager.cc:1939] T a90b206d58d04acdb1a8cb0c45d68c31 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:53.026890 1026 tablet_service.cc:1468] Processing CreateTablet for tablet cc65674750c64114b999e0e292a851f9 (DEFAULT_TABLE table=default.Table_f52b1033085542258b846453f2b91f3d [id=f05487296c954ad08343c2a49b1934ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.027238 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc65674750c64114b999e0e292a851f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.027324 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a90b206d58d04acdb1a8cb0c45d68c31 (table default.Table_43f2d7b5617b45c9a3acc51f0add89df [id=1337201a53424ea89135a36501b564d7]) successfully deleted
I20250629 01:51:53.028347 1157 tablet_replica.cc:331] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:53.028501 1157 raft_consensus.cc:2241] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.028662 1157 raft_consensus.cc:2270] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.029268 1157 ts_tablet_manager.cc:1905] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.029588 2126 raft_consensus.cc:491] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.029685 2126 raft_consensus.cc:513] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.030105 2126 leader_election.cc:290] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.030613 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cabb0298771e4fd5be29e530d3a8ed3d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.031059 988 leader_election.cc:343] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.031596 1031 tablet_service.cc:1468] Processing CreateTablet for tablet abbefa9110cc41989c33d25e3ee8ea07 (DEFAULT_TABLE table=default.Table_abe281abbab94551a471e46d3908fc39 [id=98380c6e09a445aaa12e1b73acc4ec90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.031046 1019 tablet_service.cc:1468] Processing CreateTablet for tablet dc1b24606e6f4decb30364fa4fafc147 (DEFAULT_TABLE table=default.Table_8a1e4394a537495b9bf3f9e4335f7936 [id=e6a1c068fbca47308ae288b6d28fc63d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.034713 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abbefa9110cc41989c33d25e3ee8ea07. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.035161 710 leader_election.cc:343] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 683c87ea2ad44cecb2366d4a846d9195
I20250629 01:51:53.035866 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc1b24606e6f4decb30364fa4fafc147. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.038753 1029 tablet_service.cc:1468] Processing CreateTablet for tablet db191090410941e68cf3221b9cc62f51 (DEFAULT_TABLE table=default.Table_7039f5dc37154a69870a9955aa07d568 [id=b91b10831b3b4e7893edc082174550bf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:53.039844 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11b4bdbbf5d3421ab6e3a31de90fff6d consensus state has no leader
I20250629 01:51:53.041544 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:53.042027 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db191090410941e68cf3221b9cc62f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.042981 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_36921981749241789ba5b79d109b0549"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.045248 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36921981749241789ba5b79d109b0549 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.040418 1158 ts_tablet_manager.cc:1918] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.049047 1158 log.cc:1199] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5c6f3f19a4d441f58c8c6099e04f82ec
I20250629 01:51:53.049285 1158 ts_tablet_manager.cc:1939] T 5c6f3f19a4d441f58c8c6099e04f82ec P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.051986 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3] alter complete (version 2)
I20250629 01:51:53.052140 2052 leader_election.cc:290] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.052661 1155 ts_tablet_manager.cc:1397] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.116s user 0.003s sys 0.000s
I20250629 01:51:53.053395 1155 raft_consensus.cc:357] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.053551 1155 raft_consensus.cc:383] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.053602 1155 raft_consensus.cc:738] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.053807 1154 tablet_bootstrap.cc:492] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.053782 1155 consensus_queue.cc:260] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.053910 1154 ts_tablet_manager.cc:1397] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.004s
I20250629 01:51:53.054215 1155 ts_tablet_manager.cc:1428] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:53.054469 1155 tablet_bootstrap.cc:492] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.054636 1154 raft_consensus.cc:357] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.054774 1154 raft_consensus.cc:383] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.054814 1154 raft_consensus.cc:738] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.055032 1154 consensus_queue.cc:260] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.055136 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cabb0298771e4fd5be29e530d3a8ed3d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:53.055145 710 leader_election.cc:304] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.055394 1154 ts_tablet_manager.cc:1428] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:53.055701 2131 raft_consensus.cc:2747] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:53.056771 986 leader_election.cc:343] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.056865 986 leader_election.cc:304] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.056756 1154 tablet_bootstrap.cc:492] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.058246 1154 tablet_bootstrap.cc:654] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.058521 2126 raft_consensus.cc:2747] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.058595 1155 tablet_bootstrap.cc:654] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:53.059628 712 consensus_peers.cc:489] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:53.063603 1154 tablet_bootstrap.cc:492] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.063751 1154 ts_tablet_manager.cc:1397] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:53.064276 1154 raft_consensus.cc:357] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.064433 1154 raft_consensus.cc:383] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.064502 1154 raft_consensus.cc:738] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.064579 1915 raft_consensus.cc:491] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.064689 1154 consensus_queue.cc:260] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.064692 1915 raft_consensus.cc:513] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.065176 1915 leader_election.cc:290] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.065766 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:53.066031 1154 ts_tablet_manager.cc:1428] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:53.066215 1154 tablet_bootstrap.cc:492] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.066681 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f9bbbd263fc4a41bcfc520f2d30c5e2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:53.067122 1154 tablet_bootstrap.cc:654] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.067282 850 leader_election.cc:304] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.067659 1915 raft_consensus.cc:2747] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.068758 1156 ts_tablet_manager.cc:1918] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.068892 1156 log.cc:1199] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:53.068879 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5c6f3f19a4d441f58c8c6099e04f82ec (table default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]) successfully deleted
W20250629 01:51:53.069080 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e841a0c33e504a5081bae39d731106cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e841a0c33e504a5081bae39d731106cb already in progress: creating tablet
I20250629 01:51:53.069248 1156 ts_tablet_manager.cc:1939] T a485e0e6e2f04aa6bd67a62f6d30be26 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:53.069260 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 730178bc660f4af596007721df96198f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 730178bc660f4af596007721df96198f already in progress: creating tablet
I20250629 01:51:53.070248 1156 tablet_replica.cc:331] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.070242 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a485e0e6e2f04aa6bd67a62f6d30be26 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:51:53.070477 1156 raft_consensus.cc:2241] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.072988 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]). Expected version 2 got 0
I20250629 01:51:53.073629 1156 raft_consensus.cc:2270] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.074066 1156 ts_tablet_manager.cc:1905] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:53.074932 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60e12d28ef784d898dbf9846f7063fdf consensus state has no leader
I20250629 01:51:53.075233 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af561261ab3e4a8f933991231290368e" } new_table_name: "default.Table_9790ad1c1a4447949cfe0e98c46458df" modify_external_catalogs: true
I20250629 01:51:53.078130 1159 ts_tablet_manager.cc:1428] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.089s user 0.006s sys 0.001s
I20250629 01:51:53.079420 1159 tablet_bootstrap.cc:492] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.080027 1154 tablet_bootstrap.cc:492] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.080153 1154 ts_tablet_manager.cc:1397] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250629 01:51:53.080668 1154 raft_consensus.cc:357] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.080794 1154 raft_consensus.cc:383] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.080838 1154 raft_consensus.cc:738] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.080984 1154 consensus_queue.cc:260] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.081898 1159 tablet_bootstrap.cc:654] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.082531 1154 ts_tablet_manager.cc:1428] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:53.082751 1154 tablet_bootstrap.cc:492] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.083796 1154 tablet_bootstrap.cc:654] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.093756 1157 ts_tablet_manager.cc:1918] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:53.097589 1154 tablet_bootstrap.cc:492] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.097688 1154 ts_tablet_manager.cc:1397] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20250629 01:51:53.098750 1154 raft_consensus.cc:357] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.098877 1154 raft_consensus.cc:383] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.099071 2131 consensus_queue.cc:1035] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:53.098933 1154 raft_consensus.cc:738] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.099613 1154 consensus_queue.cc:260] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.100895 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6d95a42070914b1b95a20fb55e589351"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.101239 1154 ts_tablet_manager.cc:1428] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250629 01:51:53.101357 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d95a42070914b1b95a20fb55e589351 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.101436 1154 tablet_bootstrap.cc:492] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.103399 1154 tablet_bootstrap.cc:654] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.108143 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_beded2997aef4fe0a43329e554eebfea" } new_table_name: "default.Table_c5e04a63c4a94fb086aa813f172d054a" modify_external_catalogs: true
I20250629 01:51:53.109239 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bf110bfaab6948f3a32cdf45241b31a7
I20250629 01:51:53.109889 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4db6832e616444be90ea5b21d5dace4f" } new_table_name: "default.Table_9114ba5320214ff6bb77aebc1d32ce9e" modify_external_catalogs: true
I20250629 01:51:53.110150 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet bf110bfaab6948f3a32cdf45241b31a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.110235 750 tablet_service.cc:1515] Processing DeleteTablet for tablet bf110bfaab6948f3a32cdf45241b31a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:53.110661 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet bf110bfaab6948f3a32cdf45241b31a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf110bfaab6948f3a32cdf45241b31a7 already in progress: creating tablet
I20250629 01:51:53.111075 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_119622f658584820a8fcd60b2b640e05" } modify_external_catalogs: true
I20250629 01:51:53.111219 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_119622f658584820a8fcd60b2b640e05" } modify_external_catalogs: true
I20250629 01:51:53.111435 1154 tablet_bootstrap.cc:492] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.111522 1154 ts_tablet_manager.cc:1397] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250629 01:51:53.112027 1154 raft_consensus.cc:357] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.112175 1154 raft_consensus.cc:383] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.112231 1154 raft_consensus.cc:738] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.087069 1156 ts_tablet_manager.cc:1918] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.112440 1154 consensus_queue.cc:260] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.112517 1156 log.cc:1199] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/2f7c1f1af39d47c191d34dc026f7f687
I20250629 01:51:53.112865 1154 ts_tablet_manager.cc:1428] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:53.112983 1156 ts_tablet_manager.cc:1939] T 2f7c1f1af39d47c191d34dc026f7f687 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.113071 1154 tablet_bootstrap.cc:492] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.113790 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 2f7c1f1af39d47c191d34dc026f7f687 (table default.Table_f641040c5d3f4fabb4642ae6c32b10c7 [id=7607f62687244e718732520ce1342b1a]) successfully deleted
I20250629 01:51:53.114029 2052 raft_consensus.cc:491] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.114122 2052 raft_consensus.cc:513] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.114315 1154 tablet_bootstrap.cc:654] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.114655 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 996ad157d7804856bdb9b606bd412710 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:53.114938 2052 leader_election.cc:290] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:53.115216 710 leader_election.cc:343] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8799b1858bf2408a85f0956578639393
I20250629 01:51:53.116631 1155 tablet_bootstrap.cc:492] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.118785 1155 ts_tablet_manager.cc:1397] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.002s
I20250629 01:51:53.119463 1155 raft_consensus.cc:357] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.120251 1155 raft_consensus.cc:383] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.120355 1155 raft_consensus.cc:738] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.120597 1155 consensus_queue.cc:260] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.121289 1155 ts_tablet_manager.cc:1428] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:53.121594 1155 tablet_bootstrap.cc:492] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.123124 1155 tablet_bootstrap.cc:654] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.127404 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960] alter complete (version 2)
I20250629 01:51:53.128225 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_77b9f4dc97974a79af78d31e6c953f5c" } modify_external_catalogs: true
I20250629 01:51:53.128381 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_77b9f4dc97974a79af78d31e6c953f5c" } modify_external_catalogs: true
I20250629 01:51:53.116981 1156 tablet_replica.cc:331] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.129863 1156 raft_consensus.cc:2241] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.129998 1156 raft_consensus.cc:2270] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.130407 1156 ts_tablet_manager.cc:1905] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.130841 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_415b32694788453db597034d0865a18a" } modify_external_catalogs: true
I20250629 01:51:53.131027 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_415b32694788453db597034d0865a18a" } modify_external_catalogs: true
I20250629 01:51:53.116063 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8799b1858bf2408a85f0956578639393" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.131856 712 leader_election.cc:343] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.131937 712 leader_election.cc:304] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.125299 879 tablet_service.cc:1515] Processing DeleteTablet for tablet bf110bfaab6948f3a32cdf45241b31a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.132176 2131 raft_consensus.cc:2747] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.126618 1159 tablet_bootstrap.cc:492] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.132337 1159 ts_tablet_manager.cc:1397] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.005s
I20250629 01:51:53.132916 1159 raft_consensus.cc:357] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.133059 1159 raft_consensus.cc:383] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.133113 1159 raft_consensus.cc:738] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.133852 916 raft_consensus.cc:3058] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:53.137069 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 8799b1858bf2408a85f0956578639393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.137437 1915 raft_consensus.cc:491] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.137560 1915 raft_consensus.cc:513] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:53.138029 1915 leader_election.cc:290] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:53.133294 1159 consensus_queue.cc:260] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:53.139976 850 leader_election.cc:343] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 28eaa55b3b564e2e9cf698fda98a4c8a
I20250629 01:51:53.140496 1159 ts_tablet_manager.cc:1428] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250629 01:51:53.140636 850 leader_election.cc:343] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 28eaa55b3b564e2e9cf698fda98a4c8a
I20250629 01:51:53.140746 1159 tablet_bootstrap.cc:492] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.140710 850 leader_election.cc:304] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.140873 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b9ef027c9d42048e4a6dff221df793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.141037 1158 tablet_replica.cc:331] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.141122 1915 raft_consensus.cc:2747] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.141177 1158 raft_consensus.cc:2241] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.141290 1158 raft_consensus.cc:2270] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.141956 1158 ts_tablet_manager.cc:1905] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.145051 2126 raft_consensus.cc:491] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.145170 2126 raft_consensus.cc:513] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.145747 2126 leader_election.cc:290] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.146193 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.146570 988 leader_election.cc:343] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.147346 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.147768 986 leader_election.cc:343] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.147876 986 leader_election.cc:304] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.148242 2126 raft_consensus.cc:2747] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.149459 1157 log.cc:1199] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/bedbb041e4094d8399de625badd971bb
I20250629 01:51:53.149787 1157 ts_tablet_manager.cc:1939] T bedbb041e4094d8399de625badd971bb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:53.150027 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 837) took 1118 ms. Trace:
I20250629 01:51:53.150092 1157 rpcz_store.cc:276] 0629 01:51:52.031859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.031954 (+ 95us) service_pool.cc:225] Handling call
0629 01:51:53.028310 (+996356us) ts_tablet_manager.cc:1145] Deleting tablet bedbb041e4094d8399de625badd971bb
0629 01:51:53.028318 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.091058 (+ 62740us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.093679 (+ 2621us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.150016 (+ 56337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":996216}
I20250629 01:51:53.150398 1158 ts_tablet_manager.cc:1918] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.150475 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet bedbb041e4094d8399de625badd971bb (table default.Table_c208a66223c24a10abbc5478f7469f04 [id=c8577f19a8e04dc090d81e821bf6d36c]) successfully deleted
I20250629 01:51:53.150521 1158 log.cc:1199] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/56b9ef027c9d42048e4a6dff221df793
I20250629 01:51:53.150907 1158 ts_tablet_manager.cc:1939] T 56b9ef027c9d42048e4a6dff221df793 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.151361 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 56b9ef027c9d42048e4a6dff221df793 (table default.Table_5149d5ed90ca4d6ca10f30e1694a0e1d [id=24d959060a484a2bab405fa65b92abf0]) successfully deleted
I20250629 01:51:53.152258 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 86832a3c26224c7ea0b9b4b6866d3880 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.153880 1159 tablet_bootstrap.cc:654] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.157343 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 838) took 1044 ms. Trace:
I20250629 01:51:53.158020 1155 tablet_bootstrap.cc:492] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.158694 1157 rpcz_store.cc:276] 0629 01:51:52.113220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.113812 (+ 592us) service_pool.cc:225] Handling call
0629 01:51:53.157226 (+1043414us) ts_tablet_manager.cc:1145] Deleting tablet 1336aea8f6b1436a971766d92d3c1197
0629 01:51:53.157233 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.157329 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043266}
I20250629 01:51:53.158975 1155 ts_tablet_manager.cc:1397] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
W20250629 01:51:53.159066 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb98aa9d365541969ea1dc70960760b2 consensus state has no leader
I20250629 01:51:53.158641 1159 tablet_bootstrap.cc:492] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.159245 1159 ts_tablet_manager.cc:1397] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250629 01:51:53.159816 1159 raft_consensus.cc:357] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:53.159889 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1336aea8f6b1436a971766d92d3c1197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1336aea8f6b1436a971766d92d3c1197 already in progress: creating tablet
I20250629 01:51:53.159950 1159 raft_consensus.cc:383] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.159996 1159 raft_consensus.cc:738] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.160290 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 839) took 1046 ms. Trace:
I20250629 01:51:53.160348 1157 rpcz_store.cc:276] 0629 01:51:52.114222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.114314 (+ 92us) service_pool.cc:225] Handling call
0629 01:51:53.160194 (+1045880us) ts_tablet_manager.cc:1145] Deleting tablet 2049e7465da4404a86dbaf5a77adcda6
0629 01:51:53.160201 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.160277 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1045746}
W20250629 01:51:53.160686 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2049e7465da4404a86dbaf5a77adcda6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2049e7465da4404a86dbaf5a77adcda6 already in progress: creating tablet
I20250629 01:51:53.161378 1155 raft_consensus.cc:357] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.161513 1155 raft_consensus.cc:383] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.161556 1155 raft_consensus.cc:738] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.161718 1155 consensus_queue.cc:260] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.162137 1155 ts_tablet_manager.cc:1428] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:53.163535 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b02ba93133744171884cf63996d45eb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.163653 750 tablet_service.cc:1515] Processing DeleteTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:53.163955 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b02ba93133744171884cf63996d45eb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:53.164485 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet dbfcb9233ace429db5daa593c2a1dcbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dbfcb9233ace429db5daa593c2a1dcbe already in progress: creating tablet
W20250629 01:51:53.164664 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet da93c730451d42bc8df1e582fb3af818 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da93c730451d42bc8df1e582fb3af818 already in progress: creating tablet
I20250629 01:51:53.164696 1159 consensus_queue.cc:260] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:53.164829 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c5e9646c3ad04ed580f1d3402010dd38 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5e9646c3ad04ed580f1d3402010dd38 already in progress: creating tablet
I20250629 01:51:53.165005 2131 raft_consensus.cc:491] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.165100 2131 raft_consensus.cc:513] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:53.165556 2131 leader_election.cc:290] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.165822 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d925ffce89b40258b29ec021bfa5f92" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.166652 710 leader_election.cc:343] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0d925ffce89b40258b29ec021bfa5f92
W20250629 01:51:53.166857 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet aef865f51a6b4191b2ab453c15094cc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef865f51a6b4191b2ab453c15094cc1 already in progress: creating tablet
W20250629 01:51:53.166955 712 leader_election.cc:343] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.167044 712 leader_election.cc:304] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.167212 2131 raft_consensus.cc:2747] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:53.167397 712 consensus_peers.cc:489] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:53.167656 2119 tablet.cc:1722] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:53.168417 1159 ts_tablet_manager.cc:1428] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250629 01:51:53.168498 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_53f9893ad8274a76ad3cea4fd31594c9" } modify_external_catalogs: true
I20250629 01:51:53.168640 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_53f9893ad8274a76ad3cea4fd31594c9" } modify_external_catalogs: true
W20250629 01:51:53.171110 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ef95de81b1de41d29a5db6dcdd5dd270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef95de81b1de41d29a5db6dcdd5dd270 already in progress: creating tablet
I20250629 01:51:53.171196 1915 raft_consensus.cc:491] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.171645 1915 raft_consensus.cc:513] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.172164 1915 leader_election.cc:290] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:53.172964 850 leader_election.cc:343] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 543242e4a5064268ae1e5519b569f682
W20250629 01:51:53.173171 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4a3b7d971694e15be1aaa072468dae2 consensus state has no leader
W20250629 01:51:53.173503 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 already in progress: creating tablet
I20250629 01:51:53.173480 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e5ccc8748dbe46e697a27a8368252d15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.173949 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5ccc8748dbe46e697a27a8368252d15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.175169 1159 tablet_bootstrap.cc:492] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.176023 1156 ts_tablet_manager.cc:1918] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.176226 1156 log.cc:1199] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:53.176506 1156 ts_tablet_manager.cc:1939] T 4a692f00ba244a80b9a6d07029f8d27e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.177163 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e068250f05fd4e77bf37790a635a1eb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.177567 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e068250f05fd4e77bf37790a635a1eb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:53.177897 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ee1baa976b5e47e490a0fd561b37f83d consensus state has no leader
W20250629 01:51:53.178874 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c5d45854191247bf8172d0bbfd01b6db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d45854191247bf8172d0bbfd01b6db already in progress: creating tablet
I20250629 01:51:53.179554 1159 tablet_bootstrap.cc:654] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.184962 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 996ad157d7804856bdb9b606bd412710
I20250629 01:51:53.185391 451 catalog_manager.cc:6652] ReplaceTablet: tablet 996ad157d7804856bdb9b606bd412710 deleted and replaced by tablet 64a4a34fadc44bf582fab2419a067fa5
I20250629 01:51:53.186240 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 996ad157d7804856bdb9b606bd412710 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a4a34fadc44bf582fab2419a067fa5) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.186251 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 996ad157d7804856bdb9b606bd412710 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a4a34fadc44bf582fab2419a067fa5) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.186869 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 996ad157d7804856bdb9b606bd412710 with delete_type TABLET_DATA_DELETED (replaced by tablet 64a4a34fadc44bf582fab2419a067fa5) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.187213 1157 tablet_replica.cc:331] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:53.187368 1157 raft_consensus.cc:2241] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.187512 1157 raft_consensus.cc:2270] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.188509 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 90eef69522e54b12b849d94e56355921 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:53.189760 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 996ad157d7804856bdb9b606bd412710 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996ad157d7804856bdb9b606bd412710 already in progress: creating tablet
I20250629 01:51:53.192072 1157 ts_tablet_manager.cc:1905] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.195032 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:53.195640 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7d7b8cf118eb4a5c997815b97b9b7d00
I20250629 01:51:53.196820 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d7b8cf118eb4a5c997815b97b9b7d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.198850 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.200379 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d7b8cf118eb4a5c997815b97b9b7d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.200881 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:53.200899 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 54f38ccc485043ba8b8a6f644a3e4f36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f38ccc485043ba8b8a6f644a3e4f36 already in progress: creating tablet
I20250629 01:51:53.202180 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.202395 1158 tablet_replica.cc:331] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.202548 1158 raft_consensus.cc:2241] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.202697 1158 raft_consensus.cc:2270] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.203154 1158 ts_tablet_manager.cc:1905] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.209597 1158 ts_tablet_manager.cc:1918] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:53.209759 1158 log.cc:1199] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/7d7b8cf118eb4a5c997815b97b9b7d00
I20250629 01:51:53.209721 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 77eae224763d464496066b4a98efe5ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.210177 1158 ts_tablet_manager.cc:1939] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.210388 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 69f0ae4c51924370896c624affde6761 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.211588 449 master_service.cc:948] ReplaceTablet: received request to replace tablet e304f0bfe13846f29e6390d9904a4f57 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:53.211613 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 7d7b8cf118eb4a5c997815b97b9b7d00 (table default.Table_77b9f4dc97974a79af78d31e6c953f5c [id=e6d6685d64154ba8ab80acf601e83986]) successfully deleted
I20250629 01:51:53.218739 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 90eef69522e54b12b849d94e56355921
I20250629 01:51:53.219166 452 catalog_manager.cc:6652] ReplaceTablet: tablet 90eef69522e54b12b849d94e56355921 deleted and replaced by tablet f1ad1b7de79b48058437a9dea5d30b37
I20250629 01:51:53.220235 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 90eef69522e54b12b849d94e56355921 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad1b7de79b48058437a9dea5d30b37) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.220455 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 7d7b8cf118eb4a5c997815b97b9b7d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:53.221230 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 90eef69522e54b12b849d94e56355921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90eef69522e54b12b849d94e56355921 already in progress: creating tablet
I20250629 01:51:53.221942 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 90eef69522e54b12b849d94e56355921 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad1b7de79b48058437a9dea5d30b37) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.224798 2068 log.cc:826] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Log is configured to *not* fsync() on all Append() calls
I20250629 01:51:53.225252 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e304f0bfe13846f29e6390d9904a4f57
I20250629 01:51:53.225659 449 catalog_manager.cc:6652] ReplaceTablet: tablet e304f0bfe13846f29e6390d9904a4f57 deleted and replaced by tablet 7cefc4de48e844f496459d7ae966ab52
I20250629 01:51:53.226111 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e304f0bfe13846f29e6390d9904a4f57 with delete_type TABLET_DATA_DELETED (replaced by tablet 7cefc4de48e844f496459d7ae966ab52) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.226438 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e304f0bfe13846f29e6390d9904a4f57 with delete_type TABLET_DATA_DELETED (replaced by tablet 7cefc4de48e844f496459d7ae966ab52) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:53.227269 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e304f0bfe13846f29e6390d9904a4f57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e304f0bfe13846f29e6390d9904a4f57 already in progress: creating tablet
W20250629 01:51:53.235143 850 leader_election.cc:343] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 543242e4a5064268ae1e5519b569f682
I20250629 01:51:53.235267 1154 tablet_bootstrap.cc:492] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.235247 850 leader_election.cc:304] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.235361 1154 ts_tablet_manager.cc:1397] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.122s user 0.016s sys 0.015s
I20250629 01:51:53.235450 1915 raft_consensus.cc:2747] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.235944 1154 raft_consensus.cc:357] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.236079 1154 raft_consensus.cc:383] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.236071 883 tablet_service.cc:1515] Processing DeleteTablet for tablet e304f0bfe13846f29e6390d9904a4f57 with delete_type TABLET_DATA_DELETED (replaced by tablet 7cefc4de48e844f496459d7ae966ab52) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.236145 1154 raft_consensus.cc:738] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.236290 1154 consensus_queue.cc:260] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.236649 1154 ts_tablet_manager.cc:1428] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250629 01:51:53.236879 1154 tablet_bootstrap.cc:492] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.238222 1154 tablet_bootstrap.cc:654] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.241293 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_abe281abbab94551a471e46d3908fc39" } new_table_name: "default.Table_73fffb9e284e4a57a7a64fb947efa644" modify_external_catalogs: true
I20250629 01:51:53.242513 2131 consensus_queue.cc:1035] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:53.243597 1157 ts_tablet_manager.cc:1918] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:53.243722 1157 log.cc:1199] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/cf4b52fae47f45ad9d77b0a97a854860
I20250629 01:51:53.249450 2126 raft_consensus.cc:491] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:53.249574 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abbefa9110cc41989c33d25e3ee8ea07 consensus state has no leader
I20250629 01:51:53.249558 2126 raft_consensus.cc:513] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.250041 2126 leader_election.cc:290] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.250789 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a40c1a8eec9549a8bdf695d6516440a8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.251212 986 leader_election.cc:343] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.258591 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:53.260309 1155 tablet_bootstrap.cc:492] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.260759 1157 ts_tablet_manager.cc:1939] T cf4b52fae47f45ad9d77b0a97a854860 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:53.261224 1157 tablet_replica.cc:331] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:53.261217 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]) successfully deleted
I20250629 01:51:53.261363 1157 raft_consensus.cc:2241] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.261467 1157 raft_consensus.cc:2270] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.261483 1155 tablet_bootstrap.cc:654] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:53.261620 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 12b85f220f3947ca8110f2db937e7b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12b85f220f3947ca8110f2db937e7b68 already in progress: creating tablet
I20250629 01:51:53.261852 1157 ts_tablet_manager.cc:1905] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.262447 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 867) took 1027 ms. Trace:
I20250629 01:51:53.244326 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 90eef69522e54b12b849d94e56355921 with delete_type TABLET_DATA_DELETED (replaced by tablet f1ad1b7de79b48058437a9dea5d30b37) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.262548 1156 rpcz_store.cc:276] 0629 01:51:52.234613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.234657 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:53.116944 (+882287us) ts_tablet_manager.cc:1145] Deleting tablet 4a692f00ba244a80b9a6d07029f8d27e
0629 01:51:53.116951 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.169435 (+ 52484us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.175949 (+ 6514us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.262432 (+ 86483us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":882137}
I20250629 01:51:53.251174 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a40c1a8eec9549a8bdf695d6516440a8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:53.263236 1156 tablet_replica.cc:331] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:53.263227 988 leader_election.cc:343] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.263312 988 leader_election.cc:304] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.263391 1156 raft_consensus.cc:2241] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:53.263582 2126 raft_consensus.cc:2747] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.263650 1156 raft_consensus.cc:2270] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.264066 1156 ts_tablet_manager.cc:1905] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.265633 1159 tablet_bootstrap.cc:492] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.265787 1159 ts_tablet_manager.cc:1397] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.008s
I20250629 01:51:53.266382 1159 raft_consensus.cc:357] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.266564 1159 raft_consensus.cc:383] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.266657 1159 raft_consensus.cc:738] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.266950 1159 consensus_queue.cc:260] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.268517 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4a692f00ba244a80b9a6d07029f8d27e (table default.Table_e605f1f30c8d4fdd8503b71227359798 [id=285208f892d84eea88ede0f96db5e7cf]) successfully deleted
W20250629 01:51:53.268718 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ef95de81b1de41d29a5db6dcdd5dd270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef95de81b1de41d29a5db6dcdd5dd270 already in progress: creating tablet
W20250629 01:51:53.268908 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5deb2a3d4b3a4a8d8acfbf4a31002495 already in progress: creating tablet
W20250629 01:51:53.269114 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c5d45854191247bf8172d0bbfd01b6db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5d45854191247bf8172d0bbfd01b6db already in progress: creating tablet
I20250629 01:51:53.269481 1154 tablet_bootstrap.cc:492] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.269583 1154 ts_tablet_manager.cc:1397] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.010s
I20250629 01:51:53.270154 1154 raft_consensus.cc:357] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.270242 879 tablet_service.cc:1515] Processing DeleteTablet for tablet b9b680859a1a4c56afe33f5b1b3c0793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.270298 1154 raft_consensus.cc:383] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.270345 1154 raft_consensus.cc:738] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.270527 1154 consensus_queue.cc:260] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.271062 1154 ts_tablet_manager.cc:1428] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:53.271358 1154 tablet_bootstrap.cc:492] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.272246 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]). Expected version 1 got 0
I20250629 01:51:53.272501 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_f798e82e14704a689fe84cbeb9a8d477 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 4 got 0
I20250629 01:51:53.273515 1154 tablet_bootstrap.cc:654] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:53.274256 451 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 462789e8daef42a08ff8b2cd2ddd1c49 consensus state has no leader
I20250629 01:51:53.274667 2131 raft_consensus.cc:491] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.274741 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.274760 2131 raft_consensus.cc:513] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.274999 1159 ts_tablet_manager.cc:1428] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250629 01:51:53.275128 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d with delete_type TABLET_DATA_DELETED (replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.275246 2131 leader_election.cc:290] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.275465 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.275769 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0c2dd05f5bb466fbe38ed259ce3a4b8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:53.275846 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.276686 1158 tablet_replica.cc:331] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:53.277055 712 leader_election.cc:343] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.277420 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d9c78256954c4e8191c1373909f6b086" } new_table_name: "default.Table_2ba17dbba43040089a9c2d400550f15f" modify_external_catalogs: true
I20250629 01:51:53.277963 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e354fce951943d69e4c666897f5caac with delete_type TABLET_DATA_DELETED (replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.278513 1159 tablet_bootstrap.cc:492] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.278657 1158 raft_consensus.cc:2241] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.278810 1158 raft_consensus.cc:2270] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:53.278995 710 leader_election.cc:343] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b0c2dd05f5bb466fbe38ed259ce3a4b8
I20250629 01:51:53.279079 710 leader_election.cc:304] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.278821 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bbbeb43473a4f6f892056a30f62589f with delete_type TABLET_DATA_DELETED (replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.279516 2131 raft_consensus.cc:2747] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.279819 1159 tablet_bootstrap.cc:654] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.280189 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.281749 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0f05d54fc9e649faabee85a418ebf2bb
I20250629 01:51:53.282433 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f05d54fc9e649faabee85a418ebf2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.282979 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1e4a173d9e70453c91d8ff79a710bc00
I20250629 01:51:53.283602 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2698ae66abbc4163beb2f3939554be2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.283710 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e4a173d9e70453c91d8ff79a710bc00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.285131 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e4a173d9e70453c91d8ff79a710bc00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.285557 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e4a173d9e70453c91d8ff79a710bc00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:53.286010 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2698ae66abbc4163beb2f3939554be2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.286813 1158 ts_tablet_manager.cc:1905] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.286875 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f05d54fc9e649faabee85a418ebf2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.287045 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:53.287191 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0f05d54fc9e649faabee85a418ebf2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.287850 1155 tablet_bootstrap.cc:492] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.287945 1155 ts_tablet_manager.cc:1397] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20250629 01:51:53.288527 1155 raft_consensus.cc:357] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.288654 1155 raft_consensus.cc:383] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.288697 1155 raft_consensus.cc:738] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.288882 1155 consensus_queue.cc:260] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.289274 1155 ts_tablet_manager.cc:1428] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:53.289712 1155 tablet_bootstrap.cc:492] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.292513 1155 tablet_bootstrap.cc:654] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.297631 1156 ts_tablet_manager.cc:1918] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:53.297730 1156 log.cc:1199] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/cf4b52fae47f45ad9d77b0a97a854860
I20250629 01:51:53.298056 1156 ts_tablet_manager.cc:1939] T cf4b52fae47f45ad9d77b0a97a854860 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.298969 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]) successfully deleted
W20250629 01:51:53.303217 850 consensus_peers.cc:489] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7b8cf118eb4a5c997815b97b9b7d00. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:53.304052 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b2805bb726134583891dea5763bcb7c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.304523 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2805bb726134583891dea5763bcb7c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.307696 916 raft_consensus.cc:3058] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:53.308159 1156 tablet_replica.cc:331] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.311869 2049 raft_consensus.cc:491] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.311996 2049 raft_consensus.cc:513] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.312582 2049 leader_election.cc:290] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.311287 2052 raft_consensus.cc:491] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.309916 1155 tablet_bootstrap.cc:492] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.312731 2052 raft_consensus.cc:513] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:53.312834 1155 ts_tablet_manager.cc:1397] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.002s
I20250629 01:51:53.313220 2052 leader_election.cc:290] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.313400 1155 raft_consensus.cc:357] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.313529 1155 raft_consensus.cc:383] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.313571 1155 raft_consensus.cc:738] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.313772 1155 consensus_queue.cc:260] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:53.314808 710 leader_election.cc:343] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 942f7bbe29f94083b2494aab8fbcc0c5
I20250629 01:51:53.315464 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1c1f88cfa7a84329b2ffe152996d178e" } modify_external_catalogs: true
I20250629 01:51:53.315608 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1c1f88cfa7a84329b2ffe152996d178e" } modify_external_catalogs: true
I20250629 01:51:53.316668 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:53.316743 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_29974721952f4bcba85323986edf64ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.316813 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942f7bbe29f94083b2494aab8fbcc0c5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.317184 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29974721952f4bcba85323986edf64ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:53.317224 712 consensus_peers.cc:489] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:51:53.317330 712 leader_election.cc:343] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:53.317469 712 leader_election.cc:343] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.317533 712 leader_election.cc:304] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.317821 2052 raft_consensus.cc:2747] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.318367 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9a012ad660f649da977b3123b27ecd2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.319213 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a012ad660f649da977b3123b27ecd2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.319754 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5f0f9ca3a65845f18135c7c9a9dd3991"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.320194 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f0f9ca3a65845f18135c7c9a9dd3991 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.325369 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:53.325418 1155 ts_tablet_manager.cc:1428] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250629 01:51:53.325577 1050 raft_consensus.cc:2466] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:53.325821 1155 tablet_bootstrap.cc:492] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.326071 710 leader_election.cc:304] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:53.326532 2052 raft_consensus.cc:2802] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:53.326612 2052 raft_consensus.cc:491] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:53.326678 2052 raft_consensus.cc:3058] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:53.327800 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 19b642dd58e948329442964df736b4a4
I20250629 01:51:53.327978 2126 raft_consensus.cc:491] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.328070 2126 raft_consensus.cc:513] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.328523 2126 leader_election.cc:290] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.328588 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 19b642dd58e948329442964df736b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.329639 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bbbeb43473a4f6f892056a30f62589f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:53.329921 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bbbeb43473a4f6f892056a30f62589f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.329988 988 leader_election.cc:343] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.330237 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 19b642dd58e948329442964df736b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:53.330265 986 leader_election.cc:343] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.330333 986 leader_election.cc:304] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.330617 2126 raft_consensus.cc:2747] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.312635 1156 raft_consensus.cc:2241] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.332836 1156 raft_consensus.cc:2270] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.333539 1156 ts_tablet_manager.cc:1905] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.336218 2119 tablet.cc:1722] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:53.340117 2126 raft_consensus.cc:491] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.340224 2126 raft_consensus.cc:513] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.340669 2126 leader_election.cc:290] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.341982 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.342365 988 leader_election.cc:343] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.343343 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 19b642dd58e948329442964df736b4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.343533 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.344187 986 leader_election.cc:343] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.344270 986 leader_election.cc:304] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.344687 2126 raft_consensus.cc:2747] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.345225 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fd06736c67aa446890bfcc17305ecf16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.345603 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd06736c67aa446890bfcc17305ecf16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.346359 2049 raft_consensus.cc:491] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.346457 2049 raft_consensus.cc:513] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.347018 2049 leader_election.cc:290] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.346239 2052 raft_consensus.cc:513] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:53.347795 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9f9bbbd263fc4a41bcfc520f2d30c5e2 (table default.Table_5f6743b197734f50999a12d6864e1134 [id=6ce2537a6faf4e11ab48c7fafc6451e3]). Expected version 2 got 0
I20250629 01:51:53.348779 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:53.350842 2052 leader_election.cc:290] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:53.351193 712 leader_election.cc:343] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.351791 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:53.351908 1049 raft_consensus.cc:3058] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:53.354161 1049 raft_consensus.cc:2466] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:53.354624 710 leader_election.cc:304] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:53.356196 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "241a5c9aee3f4297b67b18380ef5ef88" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.356586 712 leader_election.cc:343] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.356824 2052 raft_consensus.cc:2802] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:53.357084 2052 raft_consensus.cc:695] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:53.357383 2052 consensus_queue.cc:237] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:53.359160 710 leader_election.cc:343] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 241a5c9aee3f4297b67b18380ef5ef88
I20250629 01:51:53.359266 710 leader_election.cc:304] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.359681 2049 raft_consensus.cc:2747] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.361240 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ba33f4a793dd4e44820144e75cdceeb4" } new_table_name: "default.Table_6428b6c04c7b4884b29cdc13ccf39be2" modify_external_catalogs: true
W20250629 01:51:53.366336 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e119711a47f14100a9455a1afd84b6d1 consensus state has no leader
W20250629 01:51:53.368170 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6fa908ded8240bb962674e1efb29c74 consensus state has no leader
I20250629 01:51:53.369207 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.370167 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a205f56d1700435b9739b3651efae052"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.370556 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a205f56d1700435b9739b3651efae052 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.374593 1155 tablet_bootstrap.cc:654] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.377668 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6429e6545a1d4d5bb1b37b6799c105ad" } new_table_name: "default.Table_4bc29d1ce5c842abb7b4c888bedc32c3" modify_external_catalogs: true
I20250629 01:51:53.379011 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_09663e9d00cb47ecb9d43ed1ab066144" } new_table_name: "default.Table_bd359023e99a4eecbd2606dd52146994" modify_external_catalogs: true
I20250629 01:51:53.379470 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet ddc24d0e668c49ac825b2578849ab2a3 (table default.Table_770438a7f7324802ab8dea0697fc2423 [id=d3d515038aa84669a3a64824de88f06e]). Expected version 1 got 0
I20250629 01:51:53.380352 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:53.381702 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]). Expected version 2 got 0
I20250629 01:51:53.381984 453 catalog_manager.cc:5582] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } }
I20250629 01:51:53.382376 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
I20250629 01:51:53.382683 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 0323947b135a4308903b1eed1ca4db71 (table default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]). Expected version 3 got 0
I20250629 01:51:53.383630 1158 ts_tablet_manager.cc:1918] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.383773 1158 log.cc:1199] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a40c1a8eec9549a8bdf695d6516440a8
I20250629 01:51:53.384197 1158 ts_tablet_manager.cc:1939] T a40c1a8eec9549a8bdf695d6516440a8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.391400 879 tablet_service.cc:1468] Processing CreateTablet for tablet 58e5cb8c50734d9fbb2b09e764deb11b (DEFAULT_TABLE table=default.Table_cd7226b7561e4b56a60721679781b7a6 [id=70fdf32b444f407ca271abae608e5a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.391721 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e5cb8c50734d9fbb2b09e764deb11b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.400293 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 consensus state has no leader
I20250629 01:51:53.403820 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6864147ee415442093dab8c36960aa39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.404242 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6864147ee415442093dab8c36960aa39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.404390 1158 tablet_replica.cc:331] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.405206 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5829573f0cc6418ab66487ff0902cab5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.405344 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a40c1a8eec9549a8bdf695d6516440a8 (table default.Table_6f6d8cf186fc4934ac5690bac0754373 [id=db3232e7e81240a699e87310dc0627d3]) successfully deleted
W20250629 01:51:53.405606 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5829573f0cc6418ab66487ff0902cab5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.405882 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_edae10908319498e8780ce62b5f260c7" } modify_external_catalogs: true
I20250629 01:51:53.406047 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_edae10908319498e8780ce62b5f260c7" } modify_external_catalogs: true
I20250629 01:51:53.407150 1159 tablet_bootstrap.cc:492] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.407290 1159 ts_tablet_manager.cc:1397] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.129s user 0.011s sys 0.007s
I20250629 01:51:53.407441 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b5a50c642fa1422bb612d0edeadb4ee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.407850 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5a50c642fa1422bb612d0edeadb4ee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.408035 1159 raft_consensus.cc:357] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.408212 1159 raft_consensus.cc:383] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.408267 1159 raft_consensus.cc:738] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.408525 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_46cfbca358ce43a99a1a1dc43600e2b1" } modify_external_catalogs: true
I20250629 01:51:53.408663 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_46cfbca358ce43a99a1a1dc43600e2b1" } modify_external_catalogs: true
I20250629 01:51:53.409652 1159 consensus_queue.cc:260] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.410086 1159 ts_tablet_manager.cc:1428] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:53.410317 1159 tablet_bootstrap.cc:492] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.414722 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 0827ec622ff04ca6b2fec50e40b9e1de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:53.415306 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea12171996e6408da1af35e325233e7d has no consensus state
I20250629 01:51:53.417137 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 58e5cb8c50734d9fbb2b09e764deb11b (DEFAULT_TABLE table=default.Table_cd7226b7561e4b56a60721679781b7a6 [id=70fdf32b444f407ca271abae608e5a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.417544 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e5cb8c50734d9fbb2b09e764deb11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.418694 879 tablet_service.cc:1468] Processing CreateTablet for tablet 870dd69f67964548bd4d1575fb2cc788 (DEFAULT_TABLE table=default.Table_3b6a5b57d8964e5085b449b331a8702e [id=7574edcbfc8f45fc944c03b8f8941e7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.419001 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 870dd69f67964548bd4d1575fb2cc788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.422715 2126 raft_consensus.cc:491] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.422828 2126 raft_consensus.cc:513] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.423274 2126 leader_election.cc:290] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.423786 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d159e69137d34899ace5c245638a3c0f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.424525 986 leader_election.cc:343] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.406203 1158 raft_consensus.cc:2241] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.426649 1158 raft_consensus.cc:2270] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.427714 1158 ts_tablet_manager.cc:1905] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.428229 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d159e69137d34899ace5c245638a3c0f" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.428571 988 leader_election.cc:343] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.428650 988 leader_election.cc:304] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.428961 2126 raft_consensus.cc:2747] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.430685 1155 tablet_bootstrap.cc:492] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.430787 1155 ts_tablet_manager.cc:1397] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.002s
I20250629 01:51:53.431898 755 tablet_service.cc:1468] Processing CreateTablet for tablet 58e5cb8c50734d9fbb2b09e764deb11b (DEFAULT_TABLE table=default.Table_cd7226b7561e4b56a60721679781b7a6 [id=70fdf32b444f407ca271abae608e5a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.432031 1155 raft_consensus.cc:357] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.432174 1155 raft_consensus.cc:383] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.432204 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e5cb8c50734d9fbb2b09e764deb11b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.432216 1155 raft_consensus.cc:738] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.432402 1155 consensus_queue.cc:260] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.432474 1158 ts_tablet_manager.cc:1918] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.432554 1158 log.cc:1199] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:53.432771 1155 ts_tablet_manager.cc:1428] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:53.432787 1158 ts_tablet_manager.cc:1939] T cabb0298771e4fd5be29e530d3a8ed3d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.432996 1155 tablet_bootstrap.cc:492] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.434269 1155 tablet_bootstrap.cc:654] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.436311 2049 raft_consensus.cc:491] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.436431 2049 raft_consensus.cc:513] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.437054 2049 leader_election.cc:290] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.437466 1156 ts_tablet_manager.cc:1918] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.437727 1156 log.cc:1199] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/35990d206ba64159ab05ab7b7b19d064
I20250629 01:51:53.438719 1156 ts_tablet_manager.cc:1939] T 35990d206ba64159ab05ab7b7b19d064 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.438882 1158 tablet_replica.cc:331] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.439031 1158 raft_consensus.cc:2241] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.439145 1158 raft_consensus.cc:2270] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.440248 1158 ts_tablet_manager.cc:1905] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.440407 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "196f1bd233ab4399a9d22a55b8bc2635" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.440754 712 leader_election.cc:343] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:53.441237 710 leader_election.cc:343] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:53.441321 710 leader_election.cc:304] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.441542 1156 tablet_replica.cc:331] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.441591 2049 raft_consensus.cc:2747] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.444720 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 35990d206ba64159ab05ab7b7b19d064 (table default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 [id=8d49193b68504e77818e373ecad346b6]) successfully deleted
I20250629 01:51:53.444895 1158 ts_tablet_manager.cc:1918] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.444979 1158 log.cc:1199] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/6f3a65c9c1c740b1865b96a6e2aea524
I20250629 01:51:53.445209 1158 ts_tablet_manager.cc:1939] T 6f3a65c9c1c740b1865b96a6e2aea524 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.445237 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet cabb0298771e4fd5be29e530d3a8ed3d (table default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]) successfully deleted
I20250629 01:51:53.445372 1156 raft_consensus.cc:2241] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.445916 1156 raft_consensus.cc:2270] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.447669 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.448133 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 870dd69f67964548bd4d1575fb2cc788 (DEFAULT_TABLE table=default.Table_3b6a5b57d8964e5085b449b331a8702e [id=7574edcbfc8f45fc944c03b8f8941e7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.448554 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 870dd69f67964548bd4d1575fb2cc788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.449093 1155 tablet_bootstrap.cc:492] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.449229 1155 ts_tablet_manager.cc:1397] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250629 01:51:53.449976 1155 raft_consensus.cc:357] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.450150 1155 raft_consensus.cc:383] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.450212 1155 raft_consensus.cc:738] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.450407 1155 consensus_queue.cc:260] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.450986 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_401301df62574f26a5ca95acde130d41" } new_table_name: "default.Table_60c92ddc18dd4bab980e43d13e027a00" modify_external_catalogs: true
I20250629 01:51:53.451484 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 6f3a65c9c1c740b1865b96a6e2aea524 (table default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]) successfully deleted
I20250629 01:51:53.451455 1155 ts_tablet_manager.cc:1428] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:53.451902 1156 ts_tablet_manager.cc:1905] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.452185 1158 tablet_replica.cc:331] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.452334 1158 raft_consensus.cc:2241] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.452450 1158 raft_consensus.cc:2270] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.452834 1158 ts_tablet_manager.cc:1905] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.454869 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.458225 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a0002dca4db24405afdce3e1076eb52e
I20250629 01:51:53.458997 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a0002dca4db24405afdce3e1076eb52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.459769 1915 raft_consensus.cc:491] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.459868 1915 raft_consensus.cc:513] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.460403 1915 leader_election.cc:290] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.460845 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a0769f77ea234dd7bccc2244c08b512b
I20250629 01:51:53.461563 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a0769f77ea234dd7bccc2244c08b512b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.461800 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a0769f77ea234dd7bccc2244c08b512b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:53.461989 850 leader_election.cc:343] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e82b1352bb4d49769808dc402c85890f
I20250629 01:51:53.462173 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a0769f77ea234dd7bccc2244c08b512b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:53.462524 850 leader_election.cc:343] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e82b1352bb4d49769808dc402c85890f
I20250629 01:51:53.462637 850 leader_election.cc:304] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.462939 1915 raft_consensus.cc:2747] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.463117 1155 tablet_bootstrap.cc:492] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.463155 879 tablet_service.cc:1515] Processing DeleteTablet for tablet a0002dca4db24405afdce3e1076eb52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.464473 1158 ts_tablet_manager.cc:1918] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.464560 1158 log.cc:1199] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f6d79ac3c5f04b5db295c93c3aae92cb
I20250629 01:51:53.464813 1158 ts_tablet_manager.cc:1939] T f6d79ac3c5f04b5db295c93c3aae92cb P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.465541 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f6d79ac3c5f04b5db295c93c3aae92cb (table default.Table_7eb487cf3b984b829ea5456c141ea824 [id=16fca49978904e5c880caa5d56572c22]) successfully deleted
I20250629 01:51:53.463325 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a0002dca4db24405afdce3e1076eb52e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:53.467202 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82a837932f14419c8b32fcd2c9358215 consensus state has no leader
I20250629 01:51:53.474283 2049 consensus_queue.cc:1035] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:53.476573 1159 tablet_bootstrap.cc:654] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.476876 755 tablet_service.cc:1468] Processing CreateTablet for tablet 870dd69f67964548bd4d1575fb2cc788 (DEFAULT_TABLE table=default.Table_3b6a5b57d8964e5085b449b331a8702e [id=7574edcbfc8f45fc944c03b8f8941e7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.477253 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 870dd69f67964548bd4d1575fb2cc788. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.477722 1155 tablet_bootstrap.cc:654] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.483507 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_da7726c4aa4a4bf893ec5b19264cd6ea" } modify_external_catalogs: true
I20250629 01:51:53.483927 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_da7726c4aa4a4bf893ec5b19264cd6ea" } modify_external_catalogs: true
I20250629 01:51:53.484390 1158 tablet_replica.cc:331] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.484551 1158 raft_consensus.cc:2241] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.484684 1158 raft_consensus.cc:2270] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.484843 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2e69d514f50649f79824a1cb1da60444" } modify_external_catalogs: true
I20250629 01:51:53.485004 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2e69d514f50649f79824a1cb1da60444" } modify_external_catalogs: true
I20250629 01:51:53.485817 1158 ts_tablet_manager.cc:1905] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.491700 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 06e31520d3ad4c3395a59049755a4725
I20250629 01:51:53.493422 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e31520d3ad4c3395a59049755a4725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.493762 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e31520d3ad4c3395a59049755a4725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.493757 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 07d5eb1b3c3345c099f298df240a5c75
I20250629 01:51:53.495056 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 07d5eb1b3c3345c099f298df240a5c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.495690 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 07d5eb1b3c3345c099f298df240a5c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.496377 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5cd8e243b623461eabaa41b2b6d9988a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.496421 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 07d5eb1b3c3345c099f298df240a5c75 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:53.496804 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cd8e243b623461eabaa41b2b6d9988a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.499496 1158 ts_tablet_manager.cc:1918] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.499574 1158 log.cc:1199] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4e354fce951943d69e4c666897f5caac
I20250629 01:51:53.499800 1158 ts_tablet_manager.cc:1939] T 4e354fce951943d69e4c666897f5caac P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.500901 1159 tablet_bootstrap.cc:492] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.501019 1159 ts_tablet_manager.cc:1397] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.091s user 0.016s sys 0.000s
I20250629 01:51:53.501408 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e31520d3ad4c3395a59049755a4725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.501601 1159 raft_consensus.cc:357] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.501755 1159 raft_consensus.cc:383] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.501804 1159 raft_consensus.cc:738] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.501969 1159 consensus_queue.cc:260] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.502470 1159 ts_tablet_manager.cc:1428] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:53.502698 1159 tablet_bootstrap.cc:492] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.505987 1159 tablet_bootstrap.cc:654] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.506083 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4e354fce951943d69e4c666897f5caac (table default.Table_67f686028f0c40c991d3231bd0b0c823 [id=c51c68dd349543d4a4a8bfb1b9aa1184]) successfully deleted
I20250629 01:51:53.506325 1158 tablet_replica.cc:331] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.506484 1158 raft_consensus.cc:2241] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.506608 1158 raft_consensus.cc:2270] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.506999 1158 ts_tablet_manager.cc:1905] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.508708 2126 raft_consensus.cc:491] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.508814 2126 raft_consensus.cc:513] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.508977 1156 ts_tablet_manager.cc:1918] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.509307 2126 leader_election.cc:290] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.509491 1156 log.cc:1199] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/432680cae61946af9859760fe23110ec
I20250629 01:51:53.510037 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb4ff3220cf4d25b680235f1b4882ed" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.510609 986 leader_election.cc:343] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.510928 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eee311f8e86f4b1da839f4f16aa2b343"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.511476 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eee311f8e86f4b1da839f4f16aa2b343 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.512145 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb4ff3220cf4d25b680235f1b4882ed" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.512567 988 leader_election.cc:343] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.512657 988 leader_election.cc:304] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.512972 2126 raft_consensus.cc:2747] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.514988 1158 ts_tablet_manager.cc:1918] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.515066 1158 log.cc:1199] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9bbbeb43473a4f6f892056a30f62589f
I20250629 01:51:53.515295 1158 ts_tablet_manager.cc:1939] T 9bbbeb43473a4f6f892056a30f62589f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.516047 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9bbbeb43473a4f6f892056a30f62589f (table default.Table_20d462c5b8284f99b445c7539d756d3b [id=8705c82c65df415abf3299b10b90f48c]) successfully deleted
I20250629 01:51:53.518563 1159 tablet_bootstrap.cc:492] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.518709 1159 ts_tablet_manager.cc:1397] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250629 01:51:53.519315 1159 raft_consensus.cc:357] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.519490 1159 raft_consensus.cc:383] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.519587 1159 raft_consensus.cc:738] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.519824 1159 consensus_queue.cc:260] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.520426 1159 ts_tablet_manager.cc:1428] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:53.520699 1159 tablet_bootstrap.cc:492] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.520944 1156 ts_tablet_manager.cc:1939] T 432680cae61946af9859760fe23110ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.522212 1159 tablet_bootstrap.cc:654] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.524663 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 432680cae61946af9859760fe23110ec (table default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 [id=2dfa777b5eab4acb9adfd35ab56c17d0]) successfully deleted
I20250629 01:51:53.525650 1158 tablet_replica.cc:331] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.525686 1155 tablet_bootstrap.cc:492] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.525810 1158 raft_consensus.cc:2241] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.525861 1155 ts_tablet_manager.cc:1397] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.000s
I20250629 01:51:53.525934 1158 raft_consensus.cc:2270] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.526770 1155 raft_consensus.cc:357] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.527086 1158 ts_tablet_manager.cc:1905] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.527076 1155 raft_consensus.cc:383] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.527168 1155 raft_consensus.cc:738] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.527465 1155 consensus_queue.cc:260] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.528185 1155 ts_tablet_manager.cc:1428] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:53.528533 1155 tablet_bootstrap.cc:492] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.528612 879 tablet_service.cc:1515] Processing DeleteTablet for tablet ea00b4559e03410885180286b5e9fd10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.529704 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a" } modify_external_catalogs: true
I20250629 01:51:53.529862 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c6fb060aa64f4a3d850d4cdc08b07e3a" } modify_external_catalogs: true
I20250629 01:51:53.532395 2126 consensus_queue.cc:1035] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:53.535533 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 250ce9c0b27c40d996db138d0acb28d7
I20250629 01:51:53.536068 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 250ce9c0b27c40d996db138d0acb28d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.536521 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 250ce9c0b27c40d996db138d0acb28d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.536698 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 250ce9c0b27c40d996db138d0acb28d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.537328 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0509ea2e580b4b818e39da38211d0a23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.537902 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0509ea2e580b4b818e39da38211d0a23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.538197 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3581bbb91ff84108a35412b25174b190"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.538611 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3581bbb91ff84108a35412b25174b190 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.539934 1158 ts_tablet_manager.cc:1918] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.540674 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8a23a625023340518638020e0ba2aab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.541119 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a23a625023340518638020e0ba2aab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.541440 1158 log.cc:1199] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b6d21dfd48ea4a95a2fcecde6735974b
I20250629 01:51:53.541673 1915 raft_consensus.cc:491] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.541733 1158 ts_tablet_manager.cc:1939] T b6d21dfd48ea4a95a2fcecde6735974b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.541779 1915 raft_consensus.cc:513] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.542263 1915 leader_election.cc:290] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.542629 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b6d21dfd48ea4a95a2fcecde6735974b (table default.Table_1b37a535cd2f483a8c72aa10882a4301 [id=b516da4148384818bcc69e4259f20ef9]) successfully deleted
W20250629 01:51:53.543059 850 leader_election.cc:343] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e1fe3880e51345269e852014d0d60bb6
W20250629 01:51:53.543228 850 leader_election.cc:343] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e1fe3880e51345269e852014d0d60bb6
I20250629 01:51:53.543300 850 leader_election.cc:304] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.543640 1915 raft_consensus.cc:2747] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.545037 1159 tablet_bootstrap.cc:492] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.545176 1159 ts_tablet_manager.cc:1397] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.003s
W20250629 01:51:53.545631 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1e4a173d9e70453c91d8ff79a710bc00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e4a173d9e70453c91d8ff79a710bc00 already in progress: creating tablet
I20250629 01:51:53.546332 1159 raft_consensus.cc:357] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.546531 1159 raft_consensus.cc:383] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.546619 1159 raft_consensus.cc:738] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.546840 879 tablet_service.cc:1468] Processing CreateTablet for tablet b5bcfb2ac1674beb8e9d50d2818cec0d (DEFAULT_TABLE table=default.Table_56965d558fb9442fa7c189c772e55937 [id=1a2b3834e2294d909f399e980f12f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.546953 1159 consensus_queue.cc:260] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.547228 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcfb2ac1674beb8e9d50d2818cec0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.547369 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0f05d54fc9e649faabee85a418ebf2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f05d54fc9e649faabee85a418ebf2bb already in progress: creating tablet
I20250629 01:51:53.549795 1159 ts_tablet_manager.cc:1428] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:53.550055 1159 tablet_bootstrap.cc:492] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.546126 1156 tablet_replica.cc:331] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.552206 1156 raft_consensus.cc:2241] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:53.553310 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 19b642dd58e948329442964df736b4a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b642dd58e948329442964df736b4a4 already in progress: creating tablet
W20250629 01:51:53.554296 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a0002dca4db24405afdce3e1076eb52e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0002dca4db24405afdce3e1076eb52e already in progress: creating tablet
I20250629 01:51:53.554571 1156 raft_consensus.cc:2270] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:53.555126 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a0769f77ea234dd7bccc2244c08b512b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0769f77ea234dd7bccc2244c08b512b already in progress: creating tablet
I20250629 01:51:53.555794 1159 tablet_bootstrap.cc:654] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:53.555796 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 06e31520d3ad4c3395a59049755a4725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06e31520d3ad4c3395a59049755a4725 already in progress: creating tablet
W20250629 01:51:53.556416 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 07d5eb1b3c3345c099f298df240a5c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07d5eb1b3c3345c099f298df240a5c75 already in progress: creating tablet
W20250629 01:51:53.557039 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 250ce9c0b27c40d996db138d0acb28d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250ce9c0b27c40d996db138d0acb28d7 already in progress: creating tablet
I20250629 01:51:53.558288 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_27f4e21d29ed426089885ef45a154384"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.558369 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b5bcfb2ac1674beb8e9d50d2818cec0d (DEFAULT_TABLE table=default.Table_56965d558fb9442fa7c189c772e55937 [id=1a2b3834e2294d909f399e980f12f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.558642 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcfb2ac1674beb8e9d50d2818cec0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.558838 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27f4e21d29ed426089885ef45a154384 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.562971 2126 raft_consensus.cc:491] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.563076 2126 raft_consensus.cc:513] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.563539 2126 leader_election.cc:290] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.564033 1156 ts_tablet_manager.cc:1905] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.564181 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.564618 986 leader_election.cc:343] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.565176 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.565526 988 leader_election.cc:343] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.565609 988 leader_election.cc:304] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.565882 2126 raft_consensus.cc:2747] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.565891 1155 tablet_bootstrap.cc:654] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.567418 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0a1d504b9fd444a38d8e74bd4b2efd44" } new_table_name: "default.Table_d4e87e759e2f4bf7b03d422364f1bca2" modify_external_catalogs: true
I20250629 01:51:53.569937 1154 tablet_bootstrap.cc:492] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.570031 1154 ts_tablet_manager.cc:1397] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.299s user 0.046s sys 0.084s
I20250629 01:51:53.570585 1154 raft_consensus.cc:357] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.570704 1154 raft_consensus.cc:383] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.570741 1154 raft_consensus.cc:738] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.570988 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_06317866df174e2db51b073ad43db5db" } new_table_name: "default.Table_02a7582c640d4b0c966097f23f57cbaf" modify_external_catalogs: true
I20250629 01:51:53.571034 750 tablet_service.cc:1468] Processing CreateTablet for tablet b5bcfb2ac1674beb8e9d50d2818cec0d (DEFAULT_TABLE table=default.Table_56965d558fb9442fa7c189c772e55937 [id=1a2b3834e2294d909f399e980f12f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.571298 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5bcfb2ac1674beb8e9d50d2818cec0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.575137 1154 consensus_queue.cc:260] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:53.575512 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5a1604552122400c8fb0aca541834525 consensus state has no leader
I20250629 01:51:53.575703 1154 ts_tablet_manager.cc:1428] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:53.575915 1154 tablet_bootstrap.cc:492] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.576892 1159 tablet_bootstrap.cc:492] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.577039 1159 ts_tablet_manager.cc:1397] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20250629 01:51:53.577634 1159 raft_consensus.cc:357] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.577785 1154 tablet_bootstrap.cc:654] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.577814 1159 raft_consensus.cc:383] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.579022 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_aa4b088ab043449280d641b0ff03d845"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.579221 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d062fb830020442faee626be006926c1 consensus state has no leader
W20250629 01:51:53.579742 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4b088ab043449280d641b0ff03d845 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.579778 1915 raft_consensus.cc:491] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.580111 1915 raft_consensus.cc:513] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.580567 1915 leader_election.cc:290] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:53.581568 850 leader_election.cc:343] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0b16b6840c24484b90218fffc1da9621
W20250629 01:51:53.585170 850 leader_election.cc:343] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0b16b6840c24484b90218fffc1da9621
I20250629 01:51:53.585283 850 leader_election.cc:304] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.585320 1159 raft_consensus.cc:738] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.585623 1915 raft_consensus.cc:2747] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.585608 1159 consensus_queue.cc:260] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.588637 1159 ts_tablet_manager.cc:1428] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250629 01:51:53.588914 1159 tablet_bootstrap.cc:492] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.590430 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f7c1f1af39d47c191d34dc026f7f687 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.592070 1159 tablet_bootstrap.cc:654] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.593036 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 548674f1507743228fcd04bfd91500ae (DEFAULT_TABLE table=default.Table_681f0348d61c4df39b9e7a9806daeaca [id=c5b98260cdb64b92964547b6d11f2f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.593328 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548674f1507743228fcd04bfd91500ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.594198 1155 tablet_bootstrap.cc:492] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.592316 879 tablet_service.cc:1468] Processing CreateTablet for tablet 548674f1507743228fcd04bfd91500ae (DEFAULT_TABLE table=default.Table_681f0348d61c4df39b9e7a9806daeaca [id=c5b98260cdb64b92964547b6d11f2f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.594376 1155 ts_tablet_manager.cc:1397] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.000s
I20250629 01:51:53.594513 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548674f1507743228fcd04bfd91500ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.595276 1155 raft_consensus.cc:357] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.595458 1155 raft_consensus.cc:383] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.595548 1155 raft_consensus.cc:738] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.595760 1155 consensus_queue.cc:260] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.596349 1155 ts_tablet_manager.cc:1428] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:53.596665 1155 tablet_bootstrap.cc:492] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.601416 1156 ts_tablet_manager.cc:1918] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.601897 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5f6743b197734f50999a12d6864e1134" } new_table_name: "default.Table_75beee4f818545f98e7cdc70a5fc6df5" modify_external_catalogs: true
I20250629 01:51:53.601912 1155 tablet_bootstrap.cc:654] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.601872 1156 log.cc:1199] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e962a020664b42ea918e5952dc43aade
I20250629 01:51:53.604331 1156 ts_tablet_manager.cc:1939] T e962a020664b42ea918e5952dc43aade P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.607167 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e962a020664b42ea918e5952dc43aade (table default.Table_e0533c0c40ca4af1be6ac30ee37f3aed [id=1d5fd1e402a54d31b3fea01d38db6918]) successfully deleted
I20250629 01:51:53.607517 1156 tablet_replica.cc:331] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.607980 1156 raft_consensus.cc:2241] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.608415 1156 raft_consensus.cc:2270] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.609874 1156 ts_tablet_manager.cc:1905] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.614456 1157 ts_tablet_manager.cc:1918] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.614553 1157 log.cc:1199] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/351f4a7ae7e244999dda81d282de505d
I20250629 01:51:53.614799 1157 ts_tablet_manager.cc:1939] T 351f4a7ae7e244999dda81d282de505d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:53.616122 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 351f4a7ae7e244999dda81d282de505d (table default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]) successfully deleted
I20250629 01:51:53.616528 1157 tablet_replica.cc:331] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:53.616943 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_770438a7f7324802ab8dea0697fc2423" } new_table_name: "default.Table_e598c56d6dcd40628668325002186227" modify_external_catalogs: true
I20250629 01:51:53.619652 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 9bbbeb43473a4f6f892056a30f62589f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:53.622242 750 tablet_service.cc:1468] Processing CreateTablet for tablet 65ab6394c32640bbb7c4cd8b77686f32 (DEFAULT_TABLE table=default.Table_e3274fb78ed8400d9bc8be767b2e174c [id=40879c0375ce440283b63a93ef41a2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.622521 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ab6394c32640bbb7c4cd8b77686f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.626886 1155 tablet_bootstrap.cc:492] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.627099 1155 ts_tablet_manager.cc:1397] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
W20250629 01:51:53.627552 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddc24d0e668c49ac825b2578849ab2a3 consensus state has no leader
I20250629 01:51:53.629426 1155 raft_consensus.cc:357] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.630007 1155 raft_consensus.cc:383] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.630172 1155 raft_consensus.cc:738] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.630534 1155 consensus_queue.cc:260] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.631165 1155 ts_tablet_manager.cc:1428] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:53.631489 1155 tablet_bootstrap.cc:492] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.637202 1155 tablet_bootstrap.cc:654] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.637667 750 tablet_service.cc:1468] Processing CreateTablet for tablet eb3f46b763b24f93ace0d1327a6be240 (DEFAULT_TABLE table=default.Table_d4dacbd3dbf34f52bcb81af671ddcf6f [id=593365b6742447709be133d2ae803477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.638341 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3f46b763b24f93ace0d1327a6be240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.643435 2052 raft_consensus.cc:491] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.643607 2052 raft_consensus.cc:513] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.644567 2052 leader_election.cc:290] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:53.644869 710 leader_election.cc:343] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 1ea681bd91e54e4898bcfb72bed6198f
I20250629 01:51:53.646296 1156 ts_tablet_manager.cc:1918] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.646818 1154 tablet_bootstrap.cc:492] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:53.646806 1156 log.cc:1199] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0d925ffce89b40258b29ec021bfa5f92
I20250629 01:51:53.646973 1154 ts_tablet_manager.cc:1397] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.001s
I20250629 01:51:53.647794 1154 raft_consensus.cc:357] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.647993 1154 raft_consensus.cc:383] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.648135 1154 raft_consensus.cc:738] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.648360 1154 consensus_queue.cc:260] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.648351 1156 ts_tablet_manager.cc:1939] T 0d925ffce89b40258b29ec021bfa5f92 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.651364 1154 ts_tablet_manager.cc:1428] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:53.653162 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0d925ffce89b40258b29ec021bfa5f92 (table default.Table_1b37a535cd2f483a8c72aa10882a4301 [id=b516da4148384818bcc69e4259f20ef9]) successfully deleted
I20250629 01:51:53.653684 1156 tablet_replica.cc:331] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:53.654081 712 consensus_peers.cc:489] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:53.654443 755 tablet_service.cc:1468] Processing CreateTablet for tablet 548674f1507743228fcd04bfd91500ae (DEFAULT_TABLE table=default.Table_681f0348d61c4df39b9e7a9806daeaca [id=c5b98260cdb64b92964547b6d11f2f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.654791 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 548674f1507743228fcd04bfd91500ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.654933 1156 raft_consensus.cc:2241] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.655941 1156 raft_consensus.cc:2270] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.660466 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2474fd6ed64c40278b389fe3aabc1765" } modify_external_catalogs: true
I20250629 01:51:53.660533 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2d7d936bf3254784bd08b3abe8443a20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.660657 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2474fd6ed64c40278b389fe3aabc1765" } modify_external_catalogs: true
W20250629 01:51:53.661010 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d7d936bf3254784bd08b3abe8443a20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.662742 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_379aba83abb74b5f9264601fa00007d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.663128 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_379aba83abb74b5f9264601fa00007d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.663733 745 tablet_service.cc:1468] Processing CreateTablet for tablet 59e0b8d3a0794e29bd64cd6adbc0b908 (DEFAULT_TABLE table=default.Table_5d3980aa7f2c477889fb562e238587c6 [id=aae02be9de7445b8ba5979f69a8115c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.664002 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e0b8d3a0794e29bd64cd6adbc0b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.664292 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bf7b726d4c38474994e10c22d1aa6076" } modify_external_catalogs: true
I20250629 01:51:53.664426 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bf7b726d4c38474994e10c22d1aa6076" } modify_external_catalogs: true
I20250629 01:51:53.666195 1156 ts_tablet_manager.cc:1905] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.669554 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92068d6a585a434da503666ff8ac9536" } new_table_name: "default.Table_a5c12815f8f24b89aa5d482cde53223c" modify_external_catalogs: true
I20250629 01:51:53.670615 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c4f60a9a65f54ed781a9b1643ba15c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.671038 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c
W20250629 01:51:53.671613 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4f60a9a65f54ed781a9b1643ba15c34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.672519 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.675652 2179 tablet.cc:1722] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:53.676012 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9cfff25bea764e62954e49611b71703c
I20250629 01:51:53.676321 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:53.620303 879 tablet_service.cc:1468] Processing CreateTablet for tablet 65ab6394c32640bbb7c4cd8b77686f32 (DEFAULT_TABLE table=default.Table_e3274fb78ed8400d9bc8be767b2e174c [id=40879c0375ce440283b63a93ef41a2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.635040 2119 tablet.cc:1722] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:53.636229 883 tablet_service.cc:1468] Processing CreateTablet for tablet eb3f46b763b24f93ace0d1327a6be240 (DEFAULT_TABLE table=default.Table_d4dacbd3dbf34f52bcb81af671ddcf6f [id=593365b6742447709be133d2ae803477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.681490 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3f46b763b24f93ace0d1327a6be240. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.684240 440 catalog_manager.cc:5001] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): alter failed for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]),no further retry: Not found: Tablet not found: cf4b52fae47f45ad9d77b0a97a854860
I20250629 01:51:53.684794 2052 raft_consensus.cc:491] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.684914 2052 raft_consensus.cc:513] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.685365 2052 leader_election.cc:290] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.645385 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1ea681bd91e54e4898bcfb72bed6198f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:53.660946 877 tablet_service.cc:1468] Processing CreateTablet for tablet 59e0b8d3a0794e29bd64cd6adbc0b908 (DEFAULT_TABLE table=default.Table_5d3980aa7f2c477889fb562e238587c6 [id=aae02be9de7445b8ba5979f69a8115c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.671708 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.677379 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cfff25bea764e62954e49611b71703c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.687865 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:53.688550 1155 tablet_bootstrap.cc:492] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.688652 1155 ts_tablet_manager.cc:1397] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.006s
I20250629 01:51:53.688884 745 tablet_service.cc:1515] Processing DeleteTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 with delete_type TABLET_DATA_DELETED (replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.689265 1155 raft_consensus.cc:357] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.689391 1155 raft_consensus.cc:383] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.689429 1155 raft_consensus.cc:738] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.689584 1155 consensus_queue.cc:260] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.689953 1155 ts_tablet_manager.cc:1428] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:53.690152 1155 tablet_bootstrap.cc:492] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.691207 1156 ts_tablet_manager.cc:1918] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.691480 1156 log.cc:1199] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:53.692289 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cfff25bea764e62954e49611b71703c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.692497 1156 ts_tablet_manager.cc:1939] T 34172d05ae40417494cd8f67f4cb981e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.693377 895 tablet_service.cc:1468] Processing CreateTablet for tablet 7302021354674a19b9cb30b89270c64f (DEFAULT_TABLE table=default.Table_7a16d1646321435cb77d2338d46598f8 [id=f8463c9b218b4d75b0ee16f90dd209ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.694516 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 894) took 1020 ms. Trace:
I20250629 01:51:53.694662 1156 rpcz_store.cc:276] 0629 01:51:52.673951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.674996 (+ 1045us) service_pool.cc:225] Handling call
0629 01:51:53.653263 (+978267us) ts_tablet_manager.cc:1145] Deleting tablet 34172d05ae40417494cd8f67f4cb981e
0629 01:51:53.653271 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.683252 (+ 29981us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.690963 (+ 7711us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.694418 (+ 3455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":978161}
I20250629 01:51:53.696290 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_75f56cc1109c41f2a0a3ba83902b88ea" } modify_external_catalogs: true
I20250629 01:51:53.696437 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f1b07c24c6441f4a4965f8e055648fe" } modify_external_catalogs: true
I20250629 01:51:53.696583 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f1b07c24c6441f4a4965f8e055648fe" } modify_external_catalogs: true
W20250629 01:51:53.696887 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 95155.5us
I20250629 01:51:53.697011 1156 tablet_replica.cc:331] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.697768 890 tablet_service.cc:1468] Processing CreateTablet for tablet f567072a9aed42f997c0ea71f3f01d51 (DEFAULT_TABLE table=default.Table_cc67f532875d4d49af4ed8a1673ec007 [id=3fbda331513043d28150e501e203df25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.698256 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ab6394c32640bbb7c4cd8b77686f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.699470 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7302021354674a19b9cb30b89270c64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.700824 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8ad618a93c184b2fbee23eb637589e4a (DEFAULT_TABLE table=default.Table_0f650c0d6e344c3883bd177ecd2a699f [id=eaf0fb4d4e904915938a22cb27c7cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.701093 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad618a93c184b2fbee23eb637589e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.701648 878 tablet_service.cc:1468] Processing CreateTablet for tablet 0c422ceed7594a18b32e5f70350e5b7b (DEFAULT_TABLE table=default.Table_3610a48595354365bc24c326de4fc485 [id=961f23a5aeff4061a034c708547db086]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.701939 888 tablet_service.cc:1468] Processing CreateTablet for tablet 77f6bab8deb740cc9cf43832196e404c (DEFAULT_TABLE table=default.Table_6cbce56fa7a54b0eb7109265617d47eb [id=2c52cf9b5a7a48e2bec4991330fd7e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.702178 881 tablet_service.cc:1468] Processing CreateTablet for tablet e45d1c8068e346039524312a058e9de7 (DEFAULT_TABLE table=default.Table_b423766fabef4323aaa0a999696efb58 [id=034cacd4c60d47e4976bf2091385b468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.702204 896 tablet_service.cc:1468] Processing CreateTablet for tablet 06d591c66d734f76a1bda2194ec62776 (DEFAULT_TABLE table=default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97 [id=26a66c07c4d5463cbdd828e4980643aa]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:53.702390 712 leader_election.cc:343] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.702486 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c422ceed7594a18b32e5f70350e5b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.702471 712 leader_election.cc:304] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.702754 2052 raft_consensus.cc:2747] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.702975 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "897a4cefefe04f06b47070ec7fa1a33b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.703390 712 leader_election.cc:343] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.696439 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_75f56cc1109c41f2a0a3ba83902b88ea" } modify_external_catalogs: true
I20250629 01:51:53.697635 1156 raft_consensus.cc:2241] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.706750 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e0b8d3a0794e29bd64cd6adbc0b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.715471 1157 raft_consensus.cc:2241] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.715623 1157 raft_consensus.cc:2270] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.716156 1157 ts_tablet_manager.cc:1905] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.721159 1156 raft_consensus.cc:2270] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.722018 745 tablet_service.cc:1468] Processing CreateTablet for tablet f567072a9aed42f997c0ea71f3f01d51 (DEFAULT_TABLE table=default.Table_cc67f532875d4d49af4ed8a1673ec007 [id=3fbda331513043d28150e501e203df25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.722122 1915 raft_consensus.cc:491] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.722220 1915 raft_consensus.cc:513] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.722268 740 tablet_service.cc:1468] Processing CreateTablet for tablet 8ad618a93c184b2fbee23eb637589e4a (DEFAULT_TABLE table=default.Table_0f650c0d6e344c3883bd177ecd2a699f [id=eaf0fb4d4e904915938a22cb27c7cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.722324 750 tablet_service.cc:1468] Processing CreateTablet for tablet 7302021354674a19b9cb30b89270c64f (DEFAULT_TABLE table=default.Table_7a16d1646321435cb77d2338d46598f8 [id=f8463c9b218b4d75b0ee16f90dd209ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.722746 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:53.722735 1915 leader_election.cc:290] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:53.723448 850 leader_election.cc:343] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: b9b680859a1a4c56afe33f5b1b3c0793
I20250629 01:51:53.653885 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.724527 1158 tablet_replica.cc:331] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:53.724680 1158 raft_consensus.cc:2241] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.724795 1158 raft_consensus.cc:2270] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.724810 1156 ts_tablet_manager.cc:1905] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.725292 1158 ts_tablet_manager.cc:1905] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.731972 1158 ts_tablet_manager.cc:1918] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.732123 1158 log.cc:1199] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:53.732462 1158 ts_tablet_manager.cc:1939] T 3db333c60cb3485a9daae316b9f1a8c8 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:53.733765 1154 tablet_bootstrap.cc:492] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:53.734232 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad618a93c184b2fbee23eb637589e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.735667 1159 tablet_bootstrap.cc:492] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.735774 1159 ts_tablet_manager.cc:1397] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.147s user 0.008s sys 0.009s
I20250629 01:51:53.736258 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f567072a9aed42f997c0ea71f3f01d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.736469 1159 raft_consensus.cc:357] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.736601 1159 raft_consensus.cc:383] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.736650 1159 raft_consensus.cc:738] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.736846 1159 consensus_queue.cc:260] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.737335 1159 ts_tablet_manager.cc:1428] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:53.691569 1155 tablet_bootstrap.cc:654] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.742704 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f6bab8deb740cc9cf43832196e404c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.748438 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d591c66d734f76a1bda2194ec62776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.752200 1915 raft_consensus.cc:491] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.752285 2052 raft_consensus.cc:491] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.752301 1915 raft_consensus.cc:513] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.752415 2052 raft_consensus.cc:513] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.752781 1915 leader_election.cc:290] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.753031 2052 leader_election.cc:290] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.755348 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e45d1c8068e346039524312a058e9de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.761163 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.761576 712 leader_election.cc:343] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.762728 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f567072a9aed42f997c0ea71f3f01d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.762555 1156 ts_tablet_manager.cc:1918] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.763309 1156 log.cc:1199] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/47d4287317ec4788bd96ad174c31748d
I20250629 01:51:53.766364 1155 tablet_bootstrap.cc:492] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.766544 1155 ts_tablet_manager.cc:1397] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.002s
I20250629 01:51:53.767616 1155 raft_consensus.cc:357] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.767854 1155 raft_consensus.cc:383] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.767932 1155 raft_consensus.cc:738] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.768226 1155 consensus_queue.cc:260] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.768937 1155 ts_tablet_manager.cc:1428] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:53.769305 1155 tablet_bootstrap.cc:492] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.769645 1156 ts_tablet_manager.cc:1939] T 47d4287317ec4788bd96ad174c31748d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.772254 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 895) took 1097 ms. Trace:
I20250629 01:51:53.772919 1156 rpcz_store.cc:276] 0629 01:51:52.674047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.675125 (+ 1078us) service_pool.cc:225] Handling call
0629 01:51:53.696887 (+1021762us) ts_tablet_manager.cc:1145] Deleting tablet 47d4287317ec4788bd96ad174c31748d
0629 01:51:53.696898 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.743032 (+ 46134us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.762271 (+ 19239us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.771971 (+ 9700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021602}
I20250629 01:51:53.774852 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7302021354674a19b9cb30b89270c64f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.774899 850 leader_election.cc:343] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 77eae224763d464496066b4a98efe5ff
I20250629 01:51:53.775310 1159 tablet_bootstrap.cc:492] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.776130 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 896) took 1092 ms. Trace:
I20250629 01:51:53.776685 1159 tablet_bootstrap.cc:654] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.776783 1156 rpcz_store.cc:276] 0629 01:51:52.683699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.683886 (+ 187us) service_pool.cc:225] Handling call
0629 01:51:53.774871 (+1090985us) ts_tablet_manager.cc:1145] Deleting tablet 12b85f220f3947ca8110f2db937e7b68
0629 01:51:53.775150 (+ 279us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.775494 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090847}
I20250629 01:51:53.780558 1155 tablet_bootstrap.cc:654] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.662696 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 59e0b8d3a0794e29bd64cd6adbc0b908 (DEFAULT_TABLE table=default.Table_5d3980aa7f2c477889fb562e238587c6 [id=aae02be9de7445b8ba5979f69a8115c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.672418 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.675235 2180 tablet.cc:1722] T 9f9bbbd263fc4a41bcfc520f2d30c5e2 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:53.684808 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cfff25bea764e62954e49611b71703c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.692690 1031 tablet_service.cc:1468] Processing CreateTablet for tablet eb3f46b763b24f93ace0d1327a6be240 (DEFAULT_TABLE table=default.Table_d4dacbd3dbf34f52bcb81af671ddcf6f [id=593365b6742447709be133d2ae803477]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.790043 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e0b8d3a0794e29bd64cd6adbc0b908. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.790371 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3f46b763b24f93ace0d1327a6be240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.655612 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 65ab6394c32640bbb7c4cd8b77686f32 (DEFAULT_TABLE table=default.Table_e3274fb78ed8400d9bc8be767b2e174c [id=40879c0375ce440283b63a93ef41a2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.797498 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65ab6394c32640bbb7c4cd8b77686f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.798776 1154 tablet_bootstrap.cc:654] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.802439 1155 tablet_bootstrap.cc:492] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.802634 1155 ts_tablet_manager.cc:1397] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20250629 01:51:53.803356 1159 tablet_bootstrap.cc:492] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.803489 1159 ts_tablet_manager.cc:1397] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20250629 01:51:53.803514 1155 raft_consensus.cc:357] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.803755 1155 raft_consensus.cc:383] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.803828 1155 raft_consensus.cc:738] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.804076 1155 consensus_queue.cc:260] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.804639 1155 ts_tablet_manager.cc:1428] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:53.804606 1159 raft_consensus.cc:357] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.804877 1159 raft_consensus.cc:383] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.804971 1159 raft_consensus.cc:738] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.805177 1159 consensus_queue.cc:260] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.806696 1915 raft_consensus.cc:491] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.806800 1915 raft_consensus.cc:513] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.807245 1915 leader_election.cc:290] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.807674 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:53.808231 1159 ts_tablet_manager.cc:1428] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:53.808487 1159 tablet_bootstrap.cc:492] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.812292 1159 tablet_bootstrap.cc:654] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.812955 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5860f7100a78476b8fa1d34f6c0b8d77" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:53.818531 710 leader_election.cc:343] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 897a4cefefe04f06b47070ec7fa1a33b
I20250629 01:51:53.818629 710 leader_election.cc:304] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:53.819677 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 8ad618a93c184b2fbee23eb637589e4a (DEFAULT_TABLE table=default.Table_0f650c0d6e344c3883bd177ecd2a699f [id=eaf0fb4d4e904915938a22cb27c7cc32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.819775 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 0c422ceed7594a18b32e5f70350e5b7b (DEFAULT_TABLE table=default.Table_3610a48595354365bc24c326de4fc485 [id=961f23a5aeff4061a034c708547db086]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.819878 2052 raft_consensus.cc:2747] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.820044 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ad618a93c184b2fbee23eb637589e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.820147 2052 consensus_queue.cc:1035] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:53.820326 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c422ceed7594a18b32e5f70350e5b7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.815414 850 leader_election.cc:343] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 77eae224763d464496066b4a98efe5ff
I20250629 01:51:53.826905 850 leader_election.cc:304] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.827247 850 leader_election.cc:304] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.827515 1915 raft_consensus.cc:2747] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.820398 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:53.827679 1915 raft_consensus.cc:2747] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.827827 1050 raft_consensus.cc:2466] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:53.828173 710 leader_election.cc:304] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:53.828608 2049 raft_consensus.cc:2802] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:53.828720 2049 raft_consensus.cc:491] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:53.828833 2049 raft_consensus.cc:3058] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:53.829830 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 7302021354674a19b9cb30b89270c64f (DEFAULT_TABLE table=default.Table_7a16d1646321435cb77d2338d46598f8 [id=f8463c9b218b4d75b0ee16f90dd209ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.830235 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7302021354674a19b9cb30b89270c64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.831071 2049 raft_consensus.cc:513] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.832073 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:53.832520 712 leader_election.cc:343] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.835094 2049 leader_election.cc:290] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.837817 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:53.838145 1050 raft_consensus.cc:3058] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:53.840070 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 34172d05ae40417494cd8f67f4cb981e (table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]) successfully deleted
I20250629 01:51:53.840900 740 tablet_service.cc:1468] Processing CreateTablet for tablet 77f6bab8deb740cc9cf43832196e404c (DEFAULT_TABLE table=default.Table_6cbce56fa7a54b0eb7109265617d47eb [id=2c52cf9b5a7a48e2bec4991330fd7e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.841105 749 tablet_service.cc:1468] Processing CreateTablet for tablet 06d591c66d734f76a1bda2194ec62776 (DEFAULT_TABLE table=default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97 [id=26a66c07c4d5463cbdd828e4980643aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.841125 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 47d4287317ec4788bd96ad174c31748d (table default.Table_58a60b60dab64affa9d8b5b1acf53326 [id=d524cd9f7aba4981b939a770099325c8]) successfully deleted
W20250629 01:51:53.841439 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 12b85f220f3947ca8110f2db937e7b68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12b85f220f3947ca8110f2db937e7b68 already in progress: creating tablet
I20250629 01:51:53.841475 738 tablet_service.cc:1468] Processing CreateTablet for tablet e45d1c8068e346039524312a058e9de7 (DEFAULT_TABLE table=default.Table_b423766fabef4323aaa0a999696efb58 [id=034cacd4c60d47e4976bf2091385b468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.842135 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 06d591c66d734f76a1bda2194ec62776 (DEFAULT_TABLE table=default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97 [id=26a66c07c4d5463cbdd828e4980643aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.842296 1029 tablet_service.cc:1468] Processing CreateTablet for tablet c946c24832f9401d9fc8864bf45231ff (DEFAULT_TABLE table=default.Table_4bca012737544ca1839fbb5ab17332b7 [id=eb45c60e68b34447bb6f9de3b38a55e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.842397 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d591c66d734f76a1bda2194ec62776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.842548 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c946c24832f9401d9fc8864bf45231ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.842407 1030 tablet_service.cc:1468] Processing CreateTablet for tablet e45d1c8068e346039524312a058e9de7 (DEFAULT_TABLE table=default.Table_b423766fabef4323aaa0a999696efb58 [id=034cacd4c60d47e4976bf2091385b468]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.842948 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e45d1c8068e346039524312a058e9de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.843844 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 898) took 1116 ms. Trace:
I20250629 01:51:53.843912 1156 rpcz_store.cc:276] 0629 01:51:52.727817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.727863 (+ 46us) service_pool.cc:225] Handling call
0629 01:51:53.778800 (+1050937us) ts_tablet_manager.cc:1145] Deleting tablet 4cc7e839241e447db039b261421a7fef
0629 01:51:53.843729 (+ 64929us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.843827 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1043688}
I20250629 01:51:53.844246 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 899) took 1094 ms. Trace:
I20250629 01:51:53.844300 1156 rpcz_store.cc:276] 0629 01:51:52.749708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.749857 (+ 149us) service_pool.cc:225] Handling call
0629 01:51:53.844166 (+1094309us) ts_tablet_manager.cc:1145] Deleting tablet 59cc1690878f40e39e23e1942c75effa
0629 01:51:53.844173 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.844233 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076698}
I20250629 01:51:53.844494 1156 tablet_replica.cc:331] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.844693 1156 raft_consensus.cc:2241] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.845168 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06d591c66d734f76a1bda2194ec62776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.845218 1156 raft_consensus.cc:2270] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.845745 1156 ts_tablet_manager.cc:1905] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.847234 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e45d1c8068e346039524312a058e9de7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.845170 1155 tablet_bootstrap.cc:492] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.849514 1155 tablet_bootstrap.cc:654] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.850404 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f6bab8deb740cc9cf43832196e404c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.853140 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 3db333c60cb3485a9daae316b9f1a8c8 (table default.Table_f81f54577f464e6bbe68673617d0e0cd [id=d929c72226424c82a6d8c8bac1c3aac0]) successfully deleted
W20250629 01:51:53.853359 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c already in progress: creating tablet
W20250629 01:51:53.853556 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9cfff25bea764e62954e49611b71703c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cfff25bea764e62954e49611b71703c already in progress: creating tablet
W20250629 01:51:53.854163 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4cc7e839241e447db039b261421a7fef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cc7e839241e447db039b261421a7fef already in progress: creating tablet
W20250629 01:51:53.854362 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 59cc1690878f40e39e23e1942c75effa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59cc1690878f40e39e23e1942c75effa already in progress: creating tablet
I20250629 01:51:53.854475 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0c422ceed7594a18b32e5f70350e5b7b (DEFAULT_TABLE table=default.Table_3610a48595354365bc24c326de4fc485 [id=961f23a5aeff4061a034c708547db086]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.854823 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c422ceed7594a18b32e5f70350e5b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.856437 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d5002228f791426d81f36cbba806a047"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.861681 2179 tablet.cc:1722] T 541728ff8dc3471ba90784b4a769c8e1 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:53.862183 2180 tablet.cc:1722] T 541728ff8dc3471ba90784b4a769c8e1 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:53.863360 1156 ts_tablet_manager.cc:1918] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.863850 1156 log.cc:1199] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/241a5c9aee3f4297b67b18380ef5ef88
I20250629 01:51:53.865810 1156 ts_tablet_manager.cc:1939] T 241a5c9aee3f4297b67b18380ef5ef88 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.867842 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b" } modify_external_catalogs: true
I20250629 01:51:53.838045 2126 raft_consensus.cc:491] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.869606 2126 raft_consensus.cc:513] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.870136 2126 leader_election.cc:290] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:53.870241 1155 tablet_bootstrap.cc:492] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.870405 1155 ts_tablet_manager.cc:1397] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.005s
I20250629 01:51:53.871151 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_730ee6fd4aca4315aaf72504cecdbad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.871383 1155 raft_consensus.cc:357] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.871658 1155 raft_consensus.cc:383] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.871747 1155 raft_consensus.cc:738] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.872007 1155 consensus_queue.cc:260] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.872766 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1b3364a7417244a59e0dc1cae5101612"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:53.872895 1155 ts_tablet_manager.cc:1428] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:53.873212 1155 tablet_bootstrap.cc:492] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.874064 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:51:53.874389 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 903) took 1080 ms. Trace:
W20250629 01:51:53.875003 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5002228f791426d81f36cbba806a047 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.874783 1156 rpcz_store.cc:276] 0629 01:51:52.793548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.793596 (+ 48us) service_pool.cc:225] Handling call
0629 01:51:53.844463 (+1050867us) ts_tablet_manager.cc:1145] Deleting tablet 241a5c9aee3f4297b67b18380ef5ef88
0629 01:51:53.844470 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.850401 (+ 5931us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.862974 (+ 12573us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.874213 (+ 11239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050686}
I20250629 01:51:53.875720 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca8aea31b6c94b6da01d53b71dfeaf2b" } modify_external_catalogs: true
I20250629 01:51:53.875726 1156 tablet_replica.cc:331] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.876257 1156 raft_consensus.cc:2241] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.876641 1156 raft_consensus.cc:2270] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:53.877770 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_730ee6fd4aca4315aaf72504cecdbad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.878237 1156 ts_tablet_manager.cc:1905] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.878943 1155 tablet_bootstrap.cc:654] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:53.879359 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b3364a7417244a59e0dc1cae5101612 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.883144 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cbc70b839d4f4fd89897b14344c2b037"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.884395 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 57147.1us
W20250629 01:51:53.884639 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbc70b839d4f4fd89897b14344c2b037 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.884950 881 tablet_service.cc:1468] Processing CreateTablet for tablet c946c24832f9401d9fc8864bf45231ff (DEFAULT_TABLE table=default.Table_4bca012737544ca1839fbb5ab17332b7 [id=eb45c60e68b34447bb6f9de3b38a55e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.885447 896 tablet_service.cc:1515] Processing DeleteTablet for tablet daf312e60c014fe698d3ec9f1a1b769c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:53.885646 850 leader_election.cc:343] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b9b680859a1a4c56afe33f5b1b3c0793
I20250629 01:51:53.885746 850 leader_election.cc:304] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.886075 2193 raft_consensus.cc:2747] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.886554 1031 tablet_service.cc:1468] Processing CreateTablet for tablet f567072a9aed42f997c0ea71f3f01d51 (DEFAULT_TABLE table=default.Table_cc67f532875d4d49af4ed8a1673ec007 [id=3fbda331513043d28150e501e203df25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.886854 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f567072a9aed42f997c0ea71f3f01d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.887655 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d17573a245c247849aa573b35b93e377"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.888025 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d17573a245c247849aa573b35b93e377 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.890053 1155 tablet_bootstrap.cc:492] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.890129 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_75beee4f818545f98e7cdc70a5fc6df5 [id=6ce2537a6faf4e11ab48c7fafc6451e3] alter complete (version 3)
I20250629 01:51:53.890547 1155 ts_tablet_manager.cc:1397] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250629 01:51:53.891618 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8ea631c388eb4b60923f367cd108d236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.891957 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ea631c388eb4b60923f367cd108d236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.892948 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1912093cd0fd4042acd742fa698b5c83" } new_table_name: "default.Table_27543398d60941ce8b16591b00423498" modify_external_catalogs: true
I20250629 01:51:53.891428 1157 ts_tablet_manager.cc:1918] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.893803 1157 log.cc:1199] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/28eaa55b3b564e2e9cf698fda98a4c8a
I20250629 01:51:53.893589 1155 raft_consensus.cc:357] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.894233 1157 ts_tablet_manager.cc:1939] T 28eaa55b3b564e2e9cf698fda98a4c8a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:53.894219 1155 raft_consensus.cc:383] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.894596 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 865) took 1203 ms. Trace:
I20250629 01:51:53.894471 1155 raft_consensus.cc:738] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.894696 1157 rpcz_store.cc:276] 0629 01:51:52.690734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.690771 (+ 37us) service_pool.cc:225] Handling call
0629 01:51:53.616494 (+925723us) ts_tablet_manager.cc:1145] Deleting tablet 28eaa55b3b564e2e9cf698fda98a4c8a
0629 01:51:53.616500 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.719235 (+102735us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.891308 (+172073us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.894582 (+ 3274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24739,"tablet-delete.queue_time_us":925605}
I20250629 01:51:53.895172 1157 tablet_replica.cc:331] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:53.895382 1157 raft_consensus.cc:2241] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.895604 1157 raft_consensus.cc:2270] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.895219 1155 consensus_queue.cc:260] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:53.896260 1157 ts_tablet_manager.cc:1905] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.896781 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 28eaa55b3b564e2e9cf698fda98a4c8a (table default.Table_8b976eb77218429b9513d370b45cd5c9 [id=67701e562ee84eaf9030d915a726ebe2]) successfully deleted
I20250629 01:51:53.897693 1155 ts_tablet_manager.cc:1428] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250629 01:51:53.898836 1155 tablet_bootstrap.cc:492] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.899250 1050 raft_consensus.cc:2466] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:53.899415 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 241a5c9aee3f4297b67b18380ef5ef88 (table default.Table_9b82f19de1c74fcd9973b4a4084712e1 [id=a9617884dddf40ee883c482c170d8a22]) successfully deleted
I20250629 01:51:53.899941 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet daf312e60c014fe698d3ec9f1a1b769c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.900084 710 leader_election.cc:304] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:51:53.900633 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet daf312e60c014fe698d3ec9f1a1b769c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf312e60c014fe698d3ec9f1a1b769c already in progress: creating tablet
I20250629 01:51:53.902261 1156 ts_tablet_manager.cc:1918] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.902666 1156 log.cc:1199] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:53.903810 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 74b3badd684d45fa88dc8cbcfc61001a
I20250629 01:51:53.904696 1156 ts_tablet_manager.cc:1939] T 196f1bd233ab4399a9d22a55b8bc2635 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.906627 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:53.907330 1155 tablet_bootstrap.cc:654] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.908289 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 904) took 1114 ms. Trace:
I20250629 01:51:53.908751 1156 rpcz_store.cc:276] 0629 01:51:52.794210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.794257 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:53.875597 (+1081340us) ts_tablet_manager.cc:1145] Deleting tablet 196f1bd233ab4399a9d22a55b8bc2635
0629 01:51:53.875605 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.888935 (+ 13330us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.901730 (+ 12795us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.908167 (+ 6437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081096}
I20250629 01:51:53.909996 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 196f1bd233ab4399a9d22a55b8bc2635 (table default.Table_821e341e379740a59615270511038004 [id=910c490dcc294f58b93e5b6c4846d263]) successfully deleted
I20250629 01:51:53.910858 1156 tablet_replica.cc:331] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.911661 1156 raft_consensus.cc:2241] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.912341 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.912544 1156 raft_consensus.cc:2270] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:53.913386 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 74b3badd684d45fa88dc8cbcfc61001a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b3badd684d45fa88dc8cbcfc61001a already in progress: creating tablet
I20250629 01:51:53.914717 1156 ts_tablet_manager.cc:1905] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.915525 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.915635 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:53.915766 2193 raft_consensus.cc:491] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:53.916409 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3db333c60cb3485a9daae316b9f1a8c8 because the tablet was not found. No further retry: Not found: Tablet not found: 3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:53.916358 2193 raft_consensus.cc:513] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.916055 1159 tablet_bootstrap.cc:492] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:53.917321 1159 ts_tablet_manager.cc:1397] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.109s user 0.013s sys 0.000s
I20250629 01:51:53.918351 1159 raft_consensus.cc:357] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.918589 1159 raft_consensus.cc:383] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.918701 1159 raft_consensus.cc:738] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.918936 1159 consensus_queue.cc:260] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.920130 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d1131e0d9c06435fb732a8a72e421298
I20250629 01:51:53.920743 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5d60769b6bad4b8eb8dfa90c3047c054"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.921173 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d60769b6bad4b8eb8dfa90c3047c054 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.921357 1159 ts_tablet_manager.cc:1428] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:53.921541 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]). Expected version 3 got 0
I20250629 01:51:53.921689 1159 tablet_bootstrap.cc:492] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:53.922431 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4de24a8e2d8f4c888ab0adb848c6deed" } new_table_name: "default.Table_fbc83e65ade946eba813be6e4ff8c3f0" modify_external_catalogs: true
I20250629 01:51:53.923075 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7cf9a31f57704fea83af6665be37020e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.923462 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cf9a31f57704fea83af6665be37020e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.922603 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f4cdf270acad4465adbb179a26cedb39" } new_table_name: "default.Table_b841397a6b4b40b9ae05c1aa6d69cda1" modify_external_catalogs: true
I20250629 01:51:53.924490 1159 tablet_bootstrap.cc:654] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.924768 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_28e044c713374ff284d41c0262e1bbc2" } new_table_name: "default.Table_6428a3757bd44307b68b33e17a0eec05" modify_external_catalogs: true
W20250629 01:51:53.925339 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47cbc958eca14ed69fb45c3c2984aeef consensus state has no leader
I20250629 01:51:53.925544 2193 leader_election.cc:290] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.925634 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9200a719f8584b2899fbb70d1be7843c" } modify_external_catalogs: true
I20250629 01:51:53.925793 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9200a719f8584b2899fbb70d1be7843c" } modify_external_catalogs: true
I20250629 01:51:53.926301 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d1131e0d9c06435fb732a8a72e421298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:53.927918 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_218288faefcf4b5d9df5adbf5d4223cb" } modify_external_catalogs: true
I20250629 01:51:53.928086 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_218288faefcf4b5d9df5adbf5d4223cb" } modify_external_catalogs: true
I20250629 01:51:53.930976 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6a6e2363eabb4ce18ec597b680b149d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:53.932241 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a6e2363eabb4ce18ec597b680b149d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:53.934410 740 tablet_service.cc:1515] Processing DeleteTablet for tablet d1131e0d9c06435fb732a8a72e421298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.935201 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet d1131e0d9c06435fb732a8a72e421298 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:53.935743 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d1131e0d9c06435fb732a8a72e421298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1131e0d9c06435fb732a8a72e421298 already in progress: creating tablet
I20250629 01:51:53.938587 1155 tablet_bootstrap.cc:492] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.938740 1155 ts_tablet_manager.cc:1397] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20250629 01:51:53.938915 449 master_service.cc:948] ReplaceTablet: received request to replace tablet ec3937679b37476f934aceb30ac62d68 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:53.939741 1155 raft_consensus.cc:357] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.939960 1155 raft_consensus.cc:383] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.940066 1155 raft_consensus.cc:738] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.940358 1155 consensus_queue.cc:260] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.940855 2052 raft_consensus.cc:2802] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:53.940971 2052 raft_consensus.cc:695] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:53.941020 1155 ts_tablet_manager.cc:1428] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:53.941184 2052 consensus_queue.cc:237] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.941366 1155 tablet_bootstrap.cc:492] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.941897 755 tablet_service.cc:1515] Processing DeleteTablet for tablet daf312e60c014fe698d3ec9f1a1b769c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.942106 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.942350 750 tablet_service.cc:1515] Processing DeleteTablet for tablet f5941dd6381041538be6c92858d6ba13 with delete_type TABLET_DATA_DELETED (replaced by tablet 347fa6a12a744ee88058643b7143e39d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:53.945209 1155 tablet_bootstrap.cc:654] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.945370 750 tablet_service.cc:1468] Processing CreateTablet for tablet c946c24832f9401d9fc8864bf45231ff (DEFAULT_TABLE table=default.Table_4bca012737544ca1839fbb5ab17332b7 [id=eb45c60e68b34447bb6f9de3b38a55e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.945797 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c946c24832f9401d9fc8864bf45231ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.951208 1156 ts_tablet_manager.cc:1918] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.951377 1156 log.cc:1199] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:53.951661 745 tablet_service.cc:1468] Processing CreateTablet for tablet ea62c05f0ede4db9b48f76347e23a5cb (DEFAULT_TABLE table=default.Table_97e9884737fa43fe910d121339257f05 [id=449ad105dbda4b6e911a9a23c2af4ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.951960 1156 ts_tablet_manager.cc:1939] T e83009c1599a43d18d1c73c0101ab0be P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.951993 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea62c05f0ede4db9b48f76347e23a5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.952353 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c946c24832f9401d9fc8864bf45231ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.952476 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ea62c05f0ede4db9b48f76347e23a5cb (DEFAULT_TABLE table=default.Table_97e9884737fa43fe910d121339257f05 [id=449ad105dbda4b6e911a9a23c2af4ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.952826 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea62c05f0ede4db9b48f76347e23a5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.953030 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 905) took 1158 ms. Trace:
I20250629 01:51:53.953181 1156 rpcz_store.cc:276] 0629 01:51:52.794638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.794764 (+ 126us) service_pool.cc:225] Handling call
0629 01:51:53.910622 (+1115858us) ts_tablet_manager.cc:1145] Deleting tablet e83009c1599a43d18d1c73c0101ab0be
0629 01:51:53.910630 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.942109 (+ 31479us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.951093 (+ 8984us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.952992 (+ 1899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1095997}
I20250629 01:51:53.953742 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e83009c1599a43d18d1c73c0101ab0be (table default.Table_6b7f597bfe3947f884cf5e129049c99b [id=e4518081a6904de8a019b6d2d015d7b7]) successfully deleted
I20250629 01:51:53.954272 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 910) took 1139 ms. Trace:
I20250629 01:51:53.954461 1156 rpcz_store.cc:276] 0629 01:51:52.814306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.815094 (+ 788us) service_pool.cc:225] Handling call
0629 01:51:53.954111 (+1139017us) ts_tablet_manager.cc:1145] Deleting tablet ec3937679b37476f934aceb30ac62d68
0629 01:51:53.954119 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.954256 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138822}
W20250629 01:51:53.955451 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ec3937679b37476f934aceb30ac62d68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec3937679b37476f934aceb30ac62d68 already in progress: creating tablet
I20250629 01:51:53.956025 1156 tablet_replica.cc:331] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.956532 1156 raft_consensus.cc:2241] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.956938 1156 raft_consensus.cc:2270] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.958449 1156 ts_tablet_manager.cc:1905] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.960152 2193 raft_consensus.cc:491] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:53.960302 2193 raft_consensus.cc:513] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.960774 2193 leader_election.cc:290] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:53.964025 1155 tablet_bootstrap.cc:492] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.964183 1155 ts_tablet_manager.cc:1397] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250629 01:51:53.965061 1155 raft_consensus.cc:357] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.965281 1155 raft_consensus.cc:383] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.965363 1155 raft_consensus.cc:738] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.965605 1155 consensus_queue.cc:260] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.966167 1155 ts_tablet_manager.cc:1428] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:53.966509 1155 tablet_bootstrap.cc:492] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.968803 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:53.969486 1155 tablet_bootstrap.cc:654] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.974758 1156 ts_tablet_manager.cc:1918] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.974937 1156 log.cc:1199] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/683c87ea2ad44cecb2366d4a846d9195
I20250629 01:51:53.975629 1156 ts_tablet_manager.cc:1939] T 683c87ea2ad44cecb2366d4a846d9195 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.977655 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 906) took 1180 ms. Trace:
I20250629 01:51:53.977952 1156 rpcz_store.cc:276] 0629 01:51:52.796852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.796895 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:53.955901 (+1159006us) ts_tablet_manager.cc:1145] Deleting tablet 683c87ea2ad44cecb2366d4a846d9195
0629 01:51:53.955908 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.970779 (+ 14871us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.974671 (+ 3892us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.977609 (+ 2938us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138712}
I20250629 01:51:53.978718 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 683c87ea2ad44cecb2366d4a846d9195 (table default.Table_04cc9d29a25645f0b3facf31bd7a9f57 [id=395f2b984ff9443a90e78e8d44bca991]) successfully deleted
I20250629 01:51:53.979266 1156 tablet_replica.cc:331] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.979954 1156 raft_consensus.cc:2241] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.980268 1156 raft_consensus.cc:2270] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.981742 1156 ts_tablet_manager.cc:1905] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:53.983081 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 47cbc958eca14ed69fb45c3c2984aeef consensus state has no leader
I20250629 01:51:53.984622 1155 tablet_bootstrap.cc:492] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:53.984817 1155 ts_tablet_manager.cc:1397] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20250629 01:51:53.985700 1155 raft_consensus.cc:357] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.985838 1155 raft_consensus.cc:383] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:53.985895 1155 raft_consensus.cc:738] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:53.986043 1155 consensus_queue.cc:260] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:53.986438 1155 ts_tablet_manager.cc:1428] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:53.986665 1155 tablet_bootstrap.cc:492] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:53.988251 1155 tablet_bootstrap.cc:654] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:53.989500 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:53.989948 850 leader_election.cc:343] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 86832a3c26224c7ea0b9b4b6866d3880
I20250629 01:51:53.990387 881 tablet_service.cc:1468] Processing CreateTablet for tablet ea62c05f0ede4db9b48f76347e23a5cb (DEFAULT_TABLE table=default.Table_97e9884737fa43fe910d121339257f05 [id=449ad105dbda4b6e911a9a23c2af4ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.990731 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea62c05f0ede4db9b48f76347e23a5cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.991156 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:53.991788 850 leader_election.cc:343] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 86832a3c26224c7ea0b9b4b6866d3880
I20250629 01:51:53.991871 850 leader_election.cc:304] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.992130 1156 ts_tablet_manager.cc:1918] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:53.992352 1156 log.cc:1199] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/75ce1bd4af434f7e99092aafc0ab0118
W20250629 01:51:53.992672 988 consensus_peers.cc:489] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:53.992900 1156 ts_tablet_manager.cc:1939] T 75ce1bd4af434f7e99092aafc0ab0118 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:53.993778 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 907) took 1196 ms. Trace:
I20250629 01:51:53.993925 1156 rpcz_store.cc:276] 0629 01:51:52.797252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.797296 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:53.979144 (+1181848us) ts_tablet_manager.cc:1145] Deleting tablet 75ce1bd4af434f7e99092aafc0ab0118
0629 01:51:53.979151 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.988401 (+ 9250us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.992021 (+ 3620us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:53.993742 (+ 1721us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161866}
I20250629 01:51:53.994417 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 75ce1bd4af434f7e99092aafc0ab0118 (table default.Table_d9bb8cc1cab544148faa53db857172d3 [id=5d8dd5f1acfd4c57b7080fc663a1d32d]) successfully deleted
W20250629 01:51:53.994511 986 leader_election.cc:343] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.994773 1156 tablet_replica.cc:331] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:53.994844 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "adbb648c49e24b8085eacbe6e0a86991" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:53.995066 1156 raft_consensus.cc:2241] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:53.995215 2193 raft_consensus.cc:2747] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:53.995267 1156 raft_consensus.cc:2270] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:53.995906 1156 ts_tablet_manager.cc:1905] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:53.997586 740 tablet_service.cc:1468] Processing CreateTablet for tablet 9ea65305417e45fdb1a47768cc1ee054 (DEFAULT_TABLE table=default.Table_bb9d3aa0a3084c80a33203f45b4978eb [id=e3dc00943509471cb5142019e78cf587]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.997915 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea65305417e45fdb1a47768cc1ee054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:53.997972 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 9ea65305417e45fdb1a47768cc1ee054 (DEFAULT_TABLE table=default.Table_bb9d3aa0a3084c80a33203f45b4978eb [id=e3dc00943509471cb5142019e78cf587]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:53.998255 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea65305417e45fdb1a47768cc1ee054. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:53.999150 988 leader_election.cc:343] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:53.999229 988 leader_election.cc:304] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:53.999692 2183 raft_consensus.cc:2747] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.000638 738 tablet_service.cc:1468] Processing CreateTablet for tablet 78e379d748434664b4e2684d3f6012ab (DEFAULT_TABLE table=default.Table_620b12abd6f9476e8bf5127f618d9295 [id=688cfc291a34490ba631b108fe6f05c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.000928 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e379d748434664b4e2684d3f6012ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.002341 2049 raft_consensus.cc:491] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.002454 2049 raft_consensus.cc:513] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.003175 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dd084685cc0407da9e12b0a4df386e9" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.003611 712 leader_election.cc:343] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.003871 2049 leader_election.cc:290] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.004215 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 77f6bab8deb740cc9cf43832196e404c (DEFAULT_TABLE table=default.Table_6cbce56fa7a54b0eb7109265617d47eb [id=2c52cf9b5a7a48e2bec4991330fd7e6f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:54.004382 710 leader_election.cc:343] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8dd084685cc0407da9e12b0a4df386e9
I20250629 01:51:54.004491 710 leader_election.cc:304] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.004642 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77f6bab8deb740cc9cf43832196e404c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.005177 749 tablet_service.cc:1468] Processing CreateTablet for tablet 48028885782940d1a42186165367aa67 (DEFAULT_TABLE table=default.Table_b20baf3328ec42b1baa2585339b2e173 [id=d5cf78488efb4428944bdceeedf6c94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.005475 743 tablet_service.cc:1468] Processing CreateTablet for tablet 45461dc364d847d0b0272ac714559008 (DEFAULT_TABLE table=default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.006042 1157 ts_tablet_manager.cc:1918] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.006129 1157 log.cc:1199] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e1fe3880e51345269e852014d0d60bb6
I20250629 01:51:54.006100 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 48028885782940d1a42186165367aa67 (DEFAULT_TABLE table=default.Table_b20baf3328ec42b1baa2585339b2e173 [id=d5cf78488efb4428944bdceeedf6c94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.006108 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 78e379d748434664b4e2684d3f6012ab (DEFAULT_TABLE table=default.Table_620b12abd6f9476e8bf5127f618d9295 [id=688cfc291a34490ba631b108fe6f05c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.006392 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48028885782940d1a42186165367aa67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.006398 1157 ts_tablet_manager.cc:1939] T e1fe3880e51345269e852014d0d60bb6 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.006610 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e379d748434664b4e2684d3f6012ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.006621 2049 raft_consensus.cc:2747] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.007395 1154 tablet_bootstrap.cc:492] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.007485 1154 ts_tablet_manager.cc:1397] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.274s user 0.002s sys 0.007s
I20250629 01:51:54.008024 1154 raft_consensus.cc:357] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.008177 1154 raft_consensus.cc:383] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.008219 1154 raft_consensus.cc:738] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.008354 1154 consensus_queue.cc:260] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.008728 1154 ts_tablet_manager.cc:1428] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:54.008965 1154 tablet_bootstrap.cc:492] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.010020 1154 tablet_bootstrap.cc:654] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.012490 1155 tablet_bootstrap.cc:492] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.012624 1155 ts_tablet_manager.cc:1397] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.002s
I20250629 01:51:54.013180 1155 raft_consensus.cc:357] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.013319 1155 raft_consensus.cc:383] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.013367 1155 raft_consensus.cc:738] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.013547 1155 consensus_queue.cc:260] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.013624 2190 raft_consensus.cc:491] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.013725 2190 raft_consensus.cc:513] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.013906 1155 ts_tablet_manager.cc:1428] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:54.014166 2190 leader_election.cc:290] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.014189 2049 raft_consensus.cc:491] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.014302 2049 raft_consensus.cc:513] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.015002 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:54.015280 710 leader_election.cc:343] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f5941dd6381041538be6c92858d6ba13
I20250629 01:51:54.015816 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f5941dd6381041538be6c92858d6ba13" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.016242 712 leader_election.cc:343] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.016369 712 leader_election.cc:304] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.016549 2052 raft_consensus.cc:2747] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.016726 2049 leader_election.cc:290] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.017221 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:54.017598 986 leader_election.cc:343] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.018004 755 tablet_service.cc:1468] Processing CreateTablet for tablet 5614713dab134cb9b6d70c9a53973014 (DEFAULT_TABLE table=default.Table_04bf790258f64e6f87859f34119d5c4d [id=b882bbc93de046a48eafe21d91f8a4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.020417 2183 raft_consensus.cc:491] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.020778 2183 raft_consensus.cc:513] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.021551 2183 leader_election.cc:290] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.020354 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 5614713dab134cb9b6d70c9a53973014 (DEFAULT_TABLE table=default.Table_04bf790258f64e6f87859f34119d5c4d [id=b882bbc93de046a48eafe21d91f8a4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.023168 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5614713dab134cb9b6d70c9a53973014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.025199 1154 tablet_bootstrap.cc:492] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.025346 1154 ts_tablet_manager.cc:1397] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250629 01:51:54.025729 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da93c730451d42bc8df1e582fb3af818" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:54.026377 986 leader_election.cc:343] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.027051 1159 tablet_bootstrap.cc:492] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.027202 1159 ts_tablet_manager.cc:1397] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.001s
I20250629 01:51:54.027870 1159 raft_consensus.cc:357] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.028059 1159 raft_consensus.cc:383] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.028168 1159 raft_consensus.cc:738] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.028429 1159 consensus_queue.cc:260] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.028973 1159 ts_tablet_manager.cc:1428] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:54.029235 1159 tablet_bootstrap.cc:492] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.030783 1159 tablet_bootstrap.cc:654] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.034543 1156 ts_tablet_manager.cc:1918] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.034943 1156 log.cc:1199] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/942f7bbe29f94083b2494aab8fbcc0c5
I20250629 01:51:54.035970 1156 ts_tablet_manager.cc:1939] T 942f7bbe29f94083b2494aab8fbcc0c5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.037348 2193 raft_consensus.cc:491] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.037443 2193 raft_consensus.cc:513] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.037842 2193 leader_election.cc:290] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.042729 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48028885782940d1a42186165367aa67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.042958 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45461dc364d847d0b0272ac714559008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.042936 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 866) took 1348 ms. Trace:
I20250629 01:51:54.043022 1155 tablet_bootstrap.cc:492] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.043085 1157 rpcz_store.cc:276] 0629 01:51:52.694876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.694915 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:53.895116 (+1200201us) ts_tablet_manager.cc:1145] Deleting tablet e1fe3880e51345269e852014d0d60bb6
0629 01:51:53.895123 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:53.972064 (+ 76941us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.005974 (+ 33910us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.042924 (+ 36950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200086}
I20250629 01:51:54.043416 1157 tablet_replica.cc:331] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.043588 1157 raft_consensus.cc:2241] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.043735 1157 raft_consensus.cc:2270] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.044080 1155 tablet_bootstrap.cc:654] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.044246 1157 ts_tablet_manager.cc:1905] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.046476 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5614713dab134cb9b6d70c9a53973014. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.051841 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 908) took 1254 ms. Trace:
I20250629 01:51:54.026363 1154 raft_consensus.cc:357] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.051945 1156 rpcz_store.cc:276] 0629 01:51:52.797754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.797941 (+ 187us) service_pool.cc:225] Handling call
0629 01:51:53.994740 (+1196799us) ts_tablet_manager.cc:1145] Deleting tablet 942f7bbe29f94083b2494aab8fbcc0c5
0629 01:51:53.994747 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.025896 (+ 31149us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.034287 (+ 8391us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.051822 (+ 17535us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1176960}
I20250629 01:51:54.052136 1154 raft_consensus.cc:383] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.052292 1154 raft_consensus.cc:738] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.052358 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 909) took 1240 ms. Trace:
I20250629 01:51:54.052435 1156 rpcz_store.cc:276] 0629 01:51:52.811980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.812023 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:54.052235 (+1240212us) ts_tablet_manager.cc:1145] Deleting tablet 62369ffc1d4a4628bb24254877d6dee6
0629 01:51:54.052245 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.052343 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234433}
I20250629 01:51:54.052681 1154 consensus_queue.cc:260] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.052773 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 915) took 1165 ms. Trace:
I20250629 01:51:54.052835 1156 rpcz_store.cc:276] 0629 01:51:52.887491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.887538 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:54.052662 (+1165124us) ts_tablet_manager.cc:1145] Deleting tablet 730178bc660f4af596007721df96198f
0629 01:51:54.052671 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.052758 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164968}
I20250629 01:51:54.053146 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 920) took 1143 ms. Trace:
I20250629 01:51:54.053218 1156 rpcz_store.cc:276] 0629 01:51:52.909678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.909723 (+ 45us) service_pool.cc:225] Handling call
0629 01:51:54.053038 (+1143315us) ts_tablet_manager.cc:1145] Deleting tablet e841a0c33e504a5081bae39d731106cb
0629 01:51:54.053047 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.053132 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143159}
I20250629 01:51:54.053310 1154 ts_tablet_manager.cc:1428] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250629 01:51:54.053478 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 919) took 1144 ms. Trace:
I20250629 01:51:54.053526 1154 tablet_bootstrap.cc:492] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.053522 1156 rpcz_store.cc:276] 0629 01:51:52.909381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.920281 (+ 10900us) service_pool.cc:225] Handling call
0629 01:51:54.053406 (+1133125us) ts_tablet_manager.cc:1145] Deleting tablet 64d3efa5b59c4ae6af52c7490c316d7e
0629 01:51:54.053412 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.053470 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1132959}
I20250629 01:51:54.053752 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 918) took 1144 ms. Trace:
I20250629 01:51:54.053807 1156 rpcz_store.cc:276] 0629 01:51:52.909095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.921792 (+ 12697us) service_pool.cc:225] Handling call
0629 01:51:54.053678 (+1131886us) ts_tablet_manager.cc:1145] Deleting tablet ac9d77381f7e4241aa546eaea3077168
0629 01:51:54.053684 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.053738 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131663}
I20250629 01:51:54.054113 1156 tablet_replica.cc:331] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.054239 1156 raft_consensus.cc:2241] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.054333 1156 raft_consensus.cc:2270] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.054699 1156 ts_tablet_manager.cc:1905] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.054765 1154 tablet_bootstrap.cc:654] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.057610 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 942f7bbe29f94083b2494aab8fbcc0c5 (table default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]) successfully deleted
I20250629 01:51:54.057847 916 raft_consensus.cc:3058] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:54.057842 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 62369ffc1d4a4628bb24254877d6dee6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62369ffc1d4a4628bb24254877d6dee6 already in progress: creating tablet
W20250629 01:51:54.058476 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 730178bc660f4af596007721df96198f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 730178bc660f4af596007721df96198f already in progress: creating tablet
W20250629 01:51:54.058652 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e841a0c33e504a5081bae39d731106cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e841a0c33e504a5081bae39d731106cb already in progress: creating tablet
W20250629 01:51:54.058825 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 64d3efa5b59c4ae6af52c7490c316d7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d3efa5b59c4ae6af52c7490c316d7e already in progress: creating tablet
W20250629 01:51:54.058979 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ac9d77381f7e4241aa546eaea3077168 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9d77381f7e4241aa546eaea3077168 already in progress: creating tablet
W20250629 01:51:54.059113 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet bf110bfaab6948f3a32cdf45241b31a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf110bfaab6948f3a32cdf45241b31a7 already in progress: creating tablet
I20250629 01:51:54.063534 1156 ts_tablet_manager.cc:1918] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.063607 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e1fe3880e51345269e852014d0d60bb6 (table default.Table_cf2dc25aa17a42fb91e6903713ca0d6c [id=4e38782b406e4cd9ae3278d074a1d97e]) successfully deleted
I20250629 01:51:54.063663 1156 log.cc:1199] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8799b1858bf2408a85f0956578639393
I20250629 01:51:54.064013 1156 ts_tablet_manager.cc:1939] T 8799b1858bf2408a85f0956578639393 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.064147 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.064211 850 leader_election.cc:304] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.064232 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da93c730451d42bc8df1e582fb3af818" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.064440 2193 raft_consensus.cc:2747] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.065192 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8799b1858bf2408a85f0956578639393 (table default.Table_1b80dba0f8d04043b95a7aaa176636b9 [id=511866fda7ae4bde9cea9777b36cd76f]) successfully deleted
I20250629 01:51:54.065464 1156 tablet_replica.cc:331] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.065661 1156 raft_consensus.cc:2241] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.065845 1156 raft_consensus.cc:2270] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.066286 1156 ts_tablet_manager.cc:1905] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.070317 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 45461dc364d847d0b0272ac714559008 (DEFAULT_TABLE table=default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.070627 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45461dc364d847d0b0272ac714559008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.073884 2119 tablet.cc:1722] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:54.075129 2049 raft_consensus.cc:491] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.075225 2049 raft_consensus.cc:513] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.075881 2049 leader_election.cc:290] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.074990 1159 tablet_bootstrap.cc:492] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.076064 1159 ts_tablet_manager.cc:1397] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.001s
W20250629 01:51:54.077261 710 leader_election.cc:343] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b6d21dfd48ea4a95a2fcecde6735974b
I20250629 01:51:54.077483 1159 raft_consensus.cc:357] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.077621 1159 raft_consensus.cc:383] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.077674 1159 raft_consensus.cc:738] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.077890 1159 consensus_queue.cc:260] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.078372 1159 ts_tablet_manager.cc:1428] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:54.079248 1156 ts_tablet_manager.cc:1918] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.079371 1156 log.cc:1199] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b0c2dd05f5bb466fbe38ed259ce3a4b8
I20250629 01:51:54.079658 1156 ts_tablet_manager.cc:1939] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.080233 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6d21dfd48ea4a95a2fcecde6735974b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.080658 712 leader_election.cc:343] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.080715 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 (table default.Table_d7789b77f4814c388e1eb092297120b1 [id=b890fb8e74dc464bbf195cdfa89814be]) successfully deleted
I20250629 01:51:54.080767 712 leader_election.cc:304] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.081123 2052 raft_consensus.cc:2747] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:54.081401 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 996ad157d7804856bdb9b606bd412710 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996ad157d7804856bdb9b606bd412710 already in progress: creating tablet
I20250629 01:51:54.081676 1156 tablet_replica.cc:331] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.081900 1156 raft_consensus.cc:2241] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.082084 1156 raft_consensus.cc:2270] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.082546 1156 ts_tablet_manager.cc:1905] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.084477 1157 ts_tablet_manager.cc:1918] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.084565 1157 log.cc:1199] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/49333302a27f4ac3b76a1e492b3c1b60
I20250629 01:51:54.084795 1157 ts_tablet_manager.cc:1939] T 49333302a27f4ac3b76a1e492b3c1b60 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.086973 1154 tablet_bootstrap.cc:492] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.087097 1154 ts_tablet_manager.cc:1397] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.001s
I20250629 01:51:54.087713 1154 raft_consensus.cc:357] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.087869 1154 raft_consensus.cc:383] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.087920 1154 raft_consensus.cc:738] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.088099 1154 consensus_queue.cc:260] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.088601 1154 ts_tablet_manager.cc:1428] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:54.091094 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 59cc1690878f40e39e23e1942c75effa from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.091625 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_33af2617f1ad46a3b9a25fe7c34f4df6" } new_table_name: "default.Table_2cba6b4f427b44168c36647373065bd7" modify_external_catalogs: true
I20250629 01:51:54.092485 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1591a004d955474d8456f3f4484e8d5f" } modify_external_catalogs: true
I20250629 01:51:54.093180 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1591a004d955474d8456f3f4484e8d5f" } modify_external_catalogs: true
I20250629 01:51:54.094719 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_a5c12815f8f24b89aa5d482cde53223c [id=fd0ea1adec3e487ab647680c75b7fde8] alter complete (version 1)
I20250629 01:51:54.095846 1159 tablet_bootstrap.cc:492] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:54.064505 850 leader_election.cc:343] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 69f0ae4c51924370896c624affde6761
I20250629 01:51:54.096379 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.096416 755 tablet_service.cc:1515] Processing DeleteTablet for tablet ef976703d61442f8987ff32070a33592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.096863 896 tablet_service.cc:1468] Processing CreateTablet for tablet 9ea65305417e45fdb1a47768cc1ee054 (DEFAULT_TABLE table=default.Table_bb9d3aa0a3084c80a33203f45b4978eb [id=e3dc00943509471cb5142019e78cf587]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.096961 890 tablet_service.cc:1468] Processing CreateTablet for tablet 78e379d748434664b4e2684d3f6012ab (DEFAULT_TABLE table=default.Table_620b12abd6f9476e8bf5127f618d9295 [id=688cfc291a34490ba631b108fe6f05c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.097153 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ea65305417e45fdb1a47768cc1ee054. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.097074 895 tablet_service.cc:1468] Processing CreateTablet for tablet 45461dc364d847d0b0272ac714559008 (DEFAULT_TABLE table=default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.097339 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e379d748434664b4e2684d3f6012ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.097615 888 tablet_service.cc:1468] Processing CreateTablet for tablet 48028885782940d1a42186165367aa67 (DEFAULT_TABLE table=default.Table_b20baf3328ec42b1baa2585339b2e173 [id=d5cf78488efb4428944bdceeedf6c94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.097910 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48028885782940d1a42186165367aa67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.096375 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec with delete_type TABLET_DATA_DELETED (replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.100386 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45461dc364d847d0b0272ac714559008. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.102475 881 tablet_service.cc:1468] Processing CreateTablet for tablet 5614713dab134cb9b6d70c9a53973014 (DEFAULT_TABLE table=default.Table_04bf790258f64e6f87859f34119d5c4d [id=b882bbc93de046a48eafe21d91f8a4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.102725 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5614713dab134cb9b6d70c9a53973014. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:54.103425 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5c32f8431be4eda8c77641c20ad4d1e consensus state has no leader
W20250629 01:51:54.103737 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b4d4d6dacb84f80903522d20de4e795 consensus state has no leader
I20250629 01:51:54.104276 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 966e379adef8435887a2d8f39bb99968
I20250629 01:51:54.104853 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b
W20250629 01:51:54.104064 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab59d0a03543499fbacaf39743ca812b consensus state has no leader
I20250629 01:51:54.109376 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet ddc24d0e668c49ac825b2578849ab2a3 (table default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e]). Expected version 2 got 0
I20250629 01:51:54.109550 452 catalog_manager.cc:5582] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:54.109936 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]). Expected version 1 got 0
I20250629 01:51:54.110769 2052 raft_consensus.cc:491] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.110890 2052 raft_consensus.cc:513] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.111358 2052 leader_election.cc:290] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.113379 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.113698 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 966e379adef8435887a2d8f39bb99968 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.113747 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 999dae3757ad4c57bf4e2a92ee5203ca (DEFAULT_TABLE table=default.Table_54de7744e36d4183a8af975cdc699bd3 [id=194bdf24776b480895832c6d00becb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.113840 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 966e379adef8435887a2d8f39bb99968 (DEFAULT_TABLE table=default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.113694 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 728ab44377bc44208308df54fe5366da (DEFAULT_TABLE table=default.Table_3a7fe7d34c6e4d919869404da9508244 [id=04c90476d6b44607ac70b95142beb03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.113991 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 4b11d32f1c3e41f3b9f6d20fd837226a (DEFAULT_TABLE table=default.Table_0891a717f4824360be1c4fe4e773a511 [id=b54d23c8401a4f7e9c44c8aed5da3f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.115209 755 tablet_service.cc:1468] Processing CreateTablet for tablet 728ab44377bc44208308df54fe5366da (DEFAULT_TABLE table=default.Table_3a7fe7d34c6e4d919869404da9508244 [id=04c90476d6b44607ac70b95142beb03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.115229 738 tablet_service.cc:1468] Processing CreateTablet for tablet 966e379adef8435887a2d8f39bb99968 (DEFAULT_TABLE table=default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.115581 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728ab44377bc44208308df54fe5366da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.115558 743 tablet_service.cc:1468] Processing CreateTablet for tablet 4b11d32f1c3e41f3b9f6d20fd837226a (DEFAULT_TABLE table=default.Table_0891a717f4824360be1c4fe4e773a511 [id=b54d23c8401a4f7e9c44c8aed5da3f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.115787 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b11d32f1c3e41f3b9f6d20fd837226a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.115900 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 966e379adef8435887a2d8f39bb99968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.117283 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 966e379adef8435887a2d8f39bb99968 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.118623 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_119824cff47b4becacd02655aba89e7f" } modify_external_catalogs: true
I20250629 01:51:54.118763 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_119824cff47b4becacd02655aba89e7f" } modify_external_catalogs: true
I20250629 01:51:54.119099 888 tablet_service.cc:1468] Processing CreateTablet for tablet 4b11d32f1c3e41f3b9f6d20fd837226a (DEFAULT_TABLE table=default.Table_0891a717f4824360be1c4fe4e773a511 [id=b54d23c8401a4f7e9c44c8aed5da3f6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.119356 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b11d32f1c3e41f3b9f6d20fd837226a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:54.119626 850 leader_election.cc:343] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 69f0ae4c51924370896c624affde6761
I20250629 01:51:54.119704 850 leader_election.cc:304] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:54.120018 988 leader_election.cc:343] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.120155 988 leader_election.cc:304] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.120330 2183 raft_consensus.cc:2747] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:54.120651 988 leader_election.cc:343] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.120728 988 leader_election.cc:304] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.120890 2183 raft_consensus.cc:2747] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.121169 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.121337 2193 raft_consensus.cc:2747] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.123221 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d159e69137d34899ace5c245638a3c0f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.123368 1050 raft_consensus.cc:2466] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.123765 749 tablet_service.cc:1468] Processing CreateTablet for tablet 999dae3757ad4c57bf4e2a92ee5203ca (DEFAULT_TABLE table=default.Table_54de7744e36d4183a8af975cdc699bd3 [id=194bdf24776b480895832c6d00becb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.123983 710 leader_election.cc:304] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.124076 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999dae3757ad4c57bf4e2a92ee5203ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.124645 2052 raft_consensus.cc:2802] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.124707 2052 raft_consensus.cc:491] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.124753 2052 raft_consensus.cc:3058] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.126264 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cf553b291854a6f9e7ed78b5db5f18c with delete_type TABLET_DATA_DELETED (replaced by tablet ddc24d0e668c49ac825b2578849ab2a3) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.126814 2193 raft_consensus.cc:491] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.126914 2193 raft_consensus.cc:513] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.127382 2193 leader_election.cc:290] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.128471 2052 raft_consensus.cc:513] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.128970 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8a512c677c164b1b861a7f53d5cb8b37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.129297 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 966e379adef8435887a2d8f39bb99968 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:54.129505 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a512c677c164b1b861a7f53d5cb8b37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.129802 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999dae3757ad4c57bf4e2a92ee5203ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.129899 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d159e69137d34899ace5c245638a3c0f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.130007 1050 raft_consensus.cc:3058] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.130555 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d7d936bf3254784bd08b3abe8443a20" } modify_external_catalogs: true
I20250629 01:51:54.130683 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d7d936bf3254784bd08b3abe8443a20" } modify_external_catalogs: true
I20250629 01:51:54.131654 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f60d91fd7c1949c4ba988f3544ddb755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.132174 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f60d91fd7c1949c4ba988f3544ddb755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.132717 2052 leader_election.cc:290] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:54.132732 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b already in progress: creating tablet
I20250629 01:51:54.132900 1050 raft_consensus.cc:2466] T d159e69137d34899ace5c245638a3c0f P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:51:54.132999 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 966e379adef8435887a2d8f39bb99968 because the tablet was not found. No further retry: Not found: Tablet not found: 966e379adef8435887a2d8f39bb99968
I20250629 01:51:54.133205 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d159e69137d34899ace5c245638a3c0f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:54.133328 710 leader_election.cc:304] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:51:54.133594 712 leader_election.cc:343] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.133597 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8f48d7115c8b4e7a905249d06a3cb424"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.134028 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f48d7115c8b4e7a905249d06a3cb424 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.135203 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cf31c7a57f0b412894e2702dc66893fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.135551 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf31c7a57f0b412894e2702dc66893fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.135694 1155 tablet_bootstrap.cc:492] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.135794 1155 ts_tablet_manager.cc:1397] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.013s
I20250629 01:51:54.136382 1155 raft_consensus.cc:357] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.136489 2052 raft_consensus.cc:2802] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.136580 1155 raft_consensus.cc:383] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.136663 1155 raft_consensus.cc:738] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.136734 2052 raft_consensus.cc:695] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.136967 2052 consensus_queue.cc:237] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.136862 1155 consensus_queue.cc:260] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.138621 1155 ts_tablet_manager.cc:1428] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:54.139339 1155 tablet_bootstrap.cc:492] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.140511 1155 tablet_bootstrap.cc:654] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.141171 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 966e379adef8435887a2d8f39bb99968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.143373 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d159e69137d34899ace5c245638a3c0f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.143728 712 leader_election.cc:343] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.144280 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b11d32f1c3e41f3b9f6d20fd837226a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.147511 1159 tablet_bootstrap.cc:654] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.149219 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 989b26c3e2294005a1ebb4316c81ac29
I20250629 01:51:54.150132 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728ab44377bc44208308df54fe5366da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.150807 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 989b26c3e2294005a1ebb4316c81ac29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:54.151294 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 989b26c3e2294005a1ebb4316c81ac29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 989b26c3e2294005a1ebb4316c81ac29 already in progress: creating tablet
I20250629 01:51:54.151532 916 raft_consensus.cc:3058] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.152019 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 867) took 1456 ms. Trace:
I20250629 01:51:54.152081 1157 rpcz_store.cc:276] 0629 01:51:52.695369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.695408 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:54.043381 (+1347973us) ts_tablet_manager.cc:1145] Deleting tablet 49333302a27f4ac3b76a1e492b3c1b60
0629 01:51:54.043388 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.081587 (+ 38199us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.084414 (+ 2827us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.152006 (+ 67592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347837}
I20250629 01:51:54.153182 1157 tablet_replica.cc:331] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.153313 1157 raft_consensus.cc:2241] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.153415 1157 raft_consensus.cc:2270] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.153831 1157 ts_tablet_manager.cc:1905] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.157305 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_abdf20a022384ea199a07cc08b8b9f97" } new_table_name: "default.Table_f3aaa9e9eb034a5cbbd31b1529d0080a" modify_external_catalogs: true
I20250629 01:51:54.159372 1159 tablet_bootstrap.cc:492] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.159487 1159 ts_tablet_manager.cc:1397] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250629 01:51:54.160133 1159 raft_consensus.cc:357] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.160274 1159 raft_consensus.cc:383] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.160313 1159 raft_consensus.cc:738] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.161206 1154 tablet_bootstrap.cc:492] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.166544 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 989b26c3e2294005a1ebb4316c81ac29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.168521 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 49333302a27f4ac3b76a1e492b3c1b60 (table default.Table_3ca4f32bc0954a7c92345faedf91b471 [id=48089fc1034648639f16290cb405456a]) successfully deleted
I20250629 01:51:54.168955 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.169313 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 989b26c3e2294005a1ebb4316c81ac29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.169368 896 tablet_service.cc:1468] Processing CreateTablet for tablet 999dae3757ad4c57bf4e2a92ee5203ca (DEFAULT_TABLE table=default.Table_54de7744e36d4183a8af975cdc699bd3 [id=194bdf24776b480895832c6d00becb73]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:54.169538 850 leader_election.cc:343] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0827ec622ff04ca6b2fec50e40b9e1de
W20250629 01:51:54.169729 850 leader_election.cc:343] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0827ec622ff04ca6b2fec50e40b9e1de
I20250629 01:51:54.169835 850 leader_election.cc:304] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.170038 1915 raft_consensus.cc:2747] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.170203 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999dae3757ad4c57bf4e2a92ee5203ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.171940 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 31cee0e741584157b0333851f94c8559
I20250629 01:51:54.172348 1154 tablet_bootstrap.cc:654] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.174216 1155 tablet_bootstrap.cc:492] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.174643 1155 ts_tablet_manager.cc:1397] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250629 01:51:54.175068 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_657a566bffb64d6293ed429ab1062bb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.175540 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_657a566bffb64d6293ed429ab1062bb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.176810 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dee672b0788449d08349b1fecc061b43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.177289 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dee672b0788449d08349b1fecc061b43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.178649 1155 raft_consensus.cc:357] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.179926 1155 raft_consensus.cc:383] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.180354 1155 raft_consensus.cc:738] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.181896 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_825efeee6983439b9a45a82092f1e29e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.182371 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_825efeee6983439b9a45a82092f1e29e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.181663 1155 consensus_queue.cc:260] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.184572 881 tablet_service.cc:1468] Processing CreateTablet for tablet 966e379adef8435887a2d8f39bb99968 (DEFAULT_TABLE table=default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.184659 738 tablet_service.cc:1515] Processing DeleteTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d with delete_type TABLET_DATA_DELETED (replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.184751 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.184906 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 966e379adef8435887a2d8f39bb99968. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.184921 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.185266 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b9ef027c9d42048e4a6dff221df793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.185561 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 31cee0e741584157b0333851f94c8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.185629 1155 ts_tablet_manager.cc:1428] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
W20250629 01:51:54.186179 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 31cee0e741584157b0333851f94c8559 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31cee0e741584157b0333851f94c8559 already in progress: creating tablet
I20250629 01:51:54.187106 1155 tablet_bootstrap.cc:492] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.190301 2049 raft_consensus.cc:491] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.190407 2049 raft_consensus.cc:513] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.190872 2049 leader_election.cc:290] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.190932 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]). Expected version 2 got 0
I20250629 01:51:54.191191 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:54.191677 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4e354fce951943d69e4c666897f5caac" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.192170 712 leader_election.cc:343] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:54.192394 710 leader_election.cc:343] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 4e354fce951943d69e4c666897f5caac
I20250629 01:51:54.192481 710 leader_election.cc:304] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.192833 2049 raft_consensus.cc:2747] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.193528 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 31cee0e741584157b0333851f94c8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.194516 1155 tablet_bootstrap.cc:654] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.195312 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 343360b252574271a1f3ab20f2485258
I20250629 01:51:54.197247 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]). Expected version 2 got 0
I20250629 01:51:54.198530 888 tablet_service.cc:1468] Processing CreateTablet for tablet 728ab44377bc44208308df54fe5366da (DEFAULT_TABLE table=default.Table_3a7fe7d34c6e4d919869404da9508244 [id=04c90476d6b44607ac70b95142beb03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.198822 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 728ab44377bc44208308df54fe5366da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.201692 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2aca26af31ed4bc994f4d62603edc881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.201912 1158 tablet_replica.cc:331] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.202087 1158 raft_consensus.cc:2241] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.202217 1158 raft_consensus.cc:2270] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.203102 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet da93c730451d42bc8df1e582fb3af818 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.203282 895 tablet_service.cc:1515] Processing DeleteTablet for tablet df66984acfa44a99bf69096aff6d49ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.203342 1158 ts_tablet_manager.cc:1905] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.204241 1156 ts_tablet_manager.cc:1918] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:54.204887 1156 log.cc:1199] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/7d7b8cf118eb4a5c997815b97b9b7d00
I20250629 01:51:54.205366 1156 ts_tablet_manager.cc:1939] T 7d7b8cf118eb4a5c997815b97b9b7d00 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.208657 1158 ts_tablet_manager.cc:1918] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.208802 1158 log.cc:1199] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/2aca26af31ed4bc994f4d62603edc881
I20250629 01:51:54.209110 1158 ts_tablet_manager.cc:1939] T 2aca26af31ed4bc994f4d62603edc881 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.209954 1158 tablet_replica.cc:331] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.210156 1158 raft_consensus.cc:2241] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.210316 1158 raft_consensus.cc:2270] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.210842 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 2aca26af31ed4bc994f4d62603edc881 (table default.Table_8bf46b174da647d2b1135f050ce5e430 [id=72ebf1394e9a40faaee64e1b26e6ffe6]) successfully deleted
I20250629 01:51:54.210987 1158 ts_tablet_manager.cc:1905] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:54.212241 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e62e0683ffd4fdb80cd9646ddcc6692 consensus state has no leader
W20250629 01:51:54.213376 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ba5887dffc1460a89208447ecc6a198 consensus state has no leader
I20250629 01:51:54.215690 1158 ts_tablet_manager.cc:1918] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.215853 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 31cee0e741584157b0333851f94c8559 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:54.221011 440 catalog_manager.cc:5001] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): alter failed for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_e99d4af37fd7473394598f4c41353fd9 [id=53d17bca03a74467a681cb07cb85ef23]),no further retry: Not found: Tablet not found: cf4b52fae47f45ad9d77b0a97a854860
I20250629 01:51:54.223366 452 catalog_manager.cc:5582] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:54.227788 1915 raft_consensus.cc:491] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.227906 1915 raft_consensus.cc:513] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.228205 1159 consensus_queue.cc:260] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.228389 1915 leader_election.cc:290] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.228387 2193 raft_consensus.cc:491] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.228525 2193 raft_consensus.cc:513] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.229077 2193 leader_election.cc:290] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.216558 1154 tablet_bootstrap.cc:492] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.236222 1158 log.cc:1199] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/da93c730451d42bc8df1e582fb3af818
I20250629 01:51:54.236253 1154 ts_tablet_manager.cc:1397] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.000s
I20250629 01:51:54.236639 1158 ts_tablet_manager.cc:1939] T da93c730451d42bc8df1e582fb3af818 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.236914 1154 raft_consensus.cc:357] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.237069 1154 raft_consensus.cc:383] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.237110 1154 raft_consensus.cc:738] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.237263 1154 consensus_queue.cc:260] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.237639 1154 ts_tablet_manager.cc:1428] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:54.237738 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet da93c730451d42bc8df1e582fb3af818 (table default.Table_2e7ef5453b1945728384798f25e47b7f [id=88472d60f4244733beb9ff0fd6ce9c9f]) successfully deleted
I20250629 01:51:54.237879 1154 tablet_bootstrap.cc:492] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:54.238050 850 leader_election.cc:343] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: ea00b4559e03410885180286b5e9fd10
W20250629 01:51:54.238222 850 leader_election.cc:343] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 81b32bac5a2144d1943285f5f446d53d
W20250629 01:51:54.238356 850 leader_election.cc:343] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ea00b4559e03410885180286b5e9fd10
I20250629 01:51:54.238425 850 leader_election.cc:304] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.211587 1155 tablet_bootstrap.cc:492] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.238654 1155 ts_tablet_manager.cc:1397] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.005s
I20250629 01:51:54.238932 1154 tablet_bootstrap.cc:654] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.239239 1155 raft_consensus.cc:357] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.239374 1155 raft_consensus.cc:383] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.239427 1155 raft_consensus.cc:738] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.239578 1155 consensus_queue.cc:260] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.240026 1155 ts_tablet_manager.cc:1428] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:54.240270 1155 tablet_bootstrap.cc:492] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.241391 1155 tablet_bootstrap.cc:654] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.241472 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a692f00ba244a80b9a6d07029f8d27e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.244534 1861 raft_consensus.cc:491] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.244629 1861 raft_consensus.cc:513] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.245054 1861 leader_election.cc:290] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.245461 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "add482b533d045e7a3010064fae730cf" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.245604 1050 raft_consensus.cc:2466] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.245646 450 master_service.cc:948] ReplaceTablet: received request to replace tablet ce7a72db953646038a9d82dde04d76e4 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.246153 710 leader_election.cc:304] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.246605 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "add482b533d045e7a3010064fae730cf" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.246711 2052 raft_consensus.cc:2802] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.246773 2052 raft_consensus.cc:491] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.246829 2052 raft_consensus.cc:3058] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:54.246948 712 leader_election.cc:343] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.249660 2052 raft_consensus.cc:513] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.250962 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 936) took 1054 ms. Trace:
I20250629 01:51:54.251034 1156 rpcz_store.cc:276] 0629 01:51:53.196724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.196777 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:54.081639 (+884862us) ts_tablet_manager.cc:1145] Deleting tablet 7d7b8cf118eb4a5c997815b97b9b7d00
0629 01:51:54.081647 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.085380 (+ 3733us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.204151 (+118771us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.250948 (+ 46797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":884704}
I20250629 01:51:54.251392 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 7d7b8cf118eb4a5c997815b97b9b7d00 (table default.Table_77b9f4dc97974a79af78d31e6c953f5c [id=e6d6685d64154ba8ab80acf601e83986]) successfully deleted
I20250629 01:51:54.251848 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "add482b533d045e7a3010064fae730cf" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.251955 1159 ts_tablet_manager.cc:1428] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.092s user 0.002s sys 0.000s
I20250629 01:51:54.251969 1050 raft_consensus.cc:3058] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.252197 1159 tablet_bootstrap.cc:492] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.253268 1159 tablet_bootstrap.cc:654] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.254398 1050 raft_consensus.cc:2466] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:54.255079 2217 raft_consensus.cc:2747] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:54.255364 850 leader_election.cc:343] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:54.255448 850 leader_election.cc:304] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.255869 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 935) took 1060 ms. Trace:
I20250629 01:51:54.255935 1156 rpcz_store.cc:276] 0629 01:51:53.195552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.200844 (+ 5292us) service_pool.cc:225] Handling call
0629 01:51:54.254859 (+1054015us) ts_tablet_manager.cc:1145] Deleting tablet 54f38ccc485043ba8b8a6f644a3e4f36
0629 01:51:54.254868 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.255851 (+ 983us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053878}
W20250629 01:51:54.256284 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 54f38ccc485043ba8b8a6f644a3e4f36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f38ccc485043ba8b8a6f644a3e4f36 already in progress: creating tablet
I20250629 01:51:54.259027 1157 ts_tablet_manager.cc:1918] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.259692 1157 log.cc:1199] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9624117e97974a47ac2bfe525da6f405
I20250629 01:51:54.261422 1154 tablet_bootstrap.cc:492] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.261523 1154 ts_tablet_manager.cc:1397] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20250629 01:51:54.261848 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.262522 1155 tablet_bootstrap.cc:492] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.262620 1155 ts_tablet_manager.cc:1397] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20250629 01:51:54.262349 1157 ts_tablet_manager.cc:1939] T 9624117e97974a47ac2bfe525da6f405 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.262611 1154 raft_consensus.cc:357] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.262732 1154 raft_consensus.cc:383] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.262771 1154 raft_consensus.cc:738] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.262921 1154 consensus_queue.cc:260] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.263295 1154 ts_tablet_manager.cc:1428] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:54.263310 1155 raft_consensus.cc:357] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.263607 1154 tablet_bootstrap.cc:492] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.263602 1155 raft_consensus.cc:383] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.263660 1155 raft_consensus.cc:738] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.263803 1155 consensus_queue.cc:260] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.264201 1155 ts_tablet_manager.cc:1428] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:54.264379 2217 raft_consensus.cc:2747] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.268368 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c995ce14c9a4966b842fcba520cf87d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.268599 1158 tablet_replica.cc:331] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.268831 1158 raft_consensus.cc:2241] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.269016 1158 raft_consensus.cc:2270] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.269347 710 leader_election.cc:304] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.269205 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 868) took 1572 ms. Trace:
I20250629 01:51:54.269546 2049 raft_consensus.cc:491] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.269629 2049 raft_consensus.cc:513] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.269790 1155 tablet_bootstrap.cc:492] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.270114 2049 leader_election.cc:290] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.270223 2049 raft_consensus.cc:2802] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.270291 2049 raft_consensus.cc:695] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.270465 2049 consensus_queue.cc:237] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.271263 1158 ts_tablet_manager.cc:1905] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.271369 1154 tablet_bootstrap.cc:654] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:54.271859 712 consensus_peers.cc:489] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:54.272318 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cabb0298771e4fd5be29e530d3a8ed3d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.273972 1155 tablet_bootstrap.cc:654] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.278785 1158 ts_tablet_manager.cc:1918] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.278913 1158 log.cc:1199] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/1c995ce14c9a4966b842fcba520cf87d
I20250629 01:51:54.279172 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 81b32bac5a2144d1943285f5f446d53d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.279511 1158 ts_tablet_manager.cc:1939] T 1c995ce14c9a4966b842fcba520cf87d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:54.276042 712 leader_election.cc:343] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.269559 1157 rpcz_store.cc:276] 0629 01:51:52.695850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.695888 (+ 38us) service_pool.cc:225] Handling call
0629 01:51:54.153150 (+1457262us) ts_tablet_manager.cc:1145] Deleting tablet 9624117e97974a47ac2bfe525da6f405
0629 01:51:54.153157 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.217918 (+ 64761us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.258390 (+ 40472us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.268212 (+ 9822us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457141}
I20250629 01:51:54.283963 1157 tablet_replica.cc:331] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.284247 1157 raft_consensus.cc:2241] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.284350 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ce7a72db953646038a9d82dde04d76e4
I20250629 01:51:54.284483 1157 raft_consensus.cc:2270] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.284766 450 catalog_manager.cc:6652] ReplaceTablet: tablet ce7a72db953646038a9d82dde04d76e4 deleted and replaced by tablet d6d00ba9eb8a40358b02d85c2352fcfd
I20250629 01:51:54.285221 1157 ts_tablet_manager.cc:1905] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.285902 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ce7a72db953646038a9d82dde04d76e4 with delete_type TABLET_DATA_DELETED (replaced by tablet d6d00ba9eb8a40358b02d85c2352fcfd) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:54.286450 710 leader_election.cc:343] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:54.286513 2052 leader_election.cc:290] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.286535 710 leader_election.cc:304] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.286912 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ce7a72db953646038a9d82dde04d76e4 with delete_type TABLET_DATA_DELETED (replaced by tablet d6d00ba9eb8a40358b02d85c2352fcfd) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.287508 2052 raft_consensus.cc:2747] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.290433 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d1eb7cb229e741969597fce7379b7d40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.290937 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1eb7cb229e741969597fce7379b7d40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.291213 2052 consensus_queue.cc:1035] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:54.291899 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9624117e97974a47ac2bfe525da6f405 (table default.Table_3906fb981a8640e1906a78a321c479b2 [id=e2a731484fe346738ed611d5d7312092]) successfully deleted
I20250629 01:51:54.293115 881 tablet_service.cc:1515] Processing DeleteTablet for tablet ce7a72db953646038a9d82dde04d76e4 with delete_type TABLET_DATA_DELETED (replaced by tablet d6d00ba9eb8a40358b02d85c2352fcfd) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.296905 2193 raft_consensus.cc:491] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.297030 2193 raft_consensus.cc:513] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.297467 2193 leader_election.cc:290] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.297503 2217 raft_consensus.cc:491] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.297629 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 with delete_type TABLET_DATA_DELETED (replaced by tablet 35a62c3fe4fb4220838deece8e46d074) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.297603 2217 raft_consensus.cc:513] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.298027 2217 leader_election.cc:290] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.298573 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 1c995ce14c9a4966b842fcba520cf87d (table default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]) successfully deleted
I20250629 01:51:54.299917 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:54.300348 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ce7a72db953646038a9d82dde04d76e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce7a72db953646038a9d82dde04d76e4 already in progress: creating tablet
I20250629 01:51:54.300370 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3618e21295a84926bda02aa1d8e90a32" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:54.300891 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d0e7545edb1e4ed1b67d3167c05b0995"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.300973 1158 tablet_replica.cc:331] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:54.301322 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0e7545edb1e4ed1b67d3167c05b0995 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:54.302353 850 leader_election.cc:343] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 2f7c1f1af39d47c191d34dc026f7f687
I20250629 01:51:54.302567 1158 raft_consensus.cc:2241] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.302374 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "add482b533d045e7a3010064fae730cf" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:54.303489 850 leader_election.cc:343] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2f7c1f1af39d47c191d34dc026f7f687
I20250629 01:51:54.303615 850 leader_election.cc:304] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:54.303679 712 leader_election.cc:343] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.303872 2217 raft_consensus.cc:2747] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.303872 1158 raft_consensus.cc:2270] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.304191 850 leader_election.cc:304] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.305652 2052 raft_consensus.cc:491] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.305794 2052 raft_consensus.cc:513] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.306634 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.306692 2052 leader_election.cc:290] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.306787 1048 raft_consensus.cc:2466] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.309188 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6bc126479adc4caf8dbe1a49f8b14668" } modify_external_catalogs: true
I20250629 01:51:54.309242 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6deb0f6b87e7489d994f1b651f6b5acd" } new_table_name: "default.Table_b55682d2903c476d9754ce8cc842594c" modify_external_catalogs: true
I20250629 01:51:54.309383 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6bc126479adc4caf8dbe1a49f8b14668" } modify_external_catalogs: true
I20250629 01:51:54.309340 1158 ts_tablet_manager.cc:1905] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.310441 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_254111a6f5a74f2c9f3b57c27da3b8a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.310881 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_254111a6f5a74f2c9f3b57c27da3b8a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.312199 710 leader_election.cc:304] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.312191 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.312245 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8b1d03de93bf48a1a4f8ed74a7ac56c3" } modify_external_catalogs: true
I20250629 01:51:54.312374 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8b1d03de93bf48a1a4f8ed74a7ac56c3" } modify_external_catalogs: true
W20250629 01:51:54.312538 712 leader_election.cc:343] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.312578 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e99d4af37fd7473394598f4c41353fd9" } new_table_name: "default.Table_72800b006f6e4b2eab6c57ecea71aef0" modify_external_catalogs: true
I20250629 01:51:54.313432 2052 raft_consensus.cc:2802] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.313495 2052 raft_consensus.cc:491] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.313556 2052 raft_consensus.cc:3058] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.319252 2052 raft_consensus.cc:513] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.319399 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f4cf919a96f34167b7dc9c3dfe7946a4
I20250629 01:51:54.320276 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f4cf919a96f34167b7dc9c3dfe7946a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.320050 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:54.320640 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet f4cf919a96f34167b7dc9c3dfe7946a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:54.321964 712 leader_election.cc:343] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.322510 2052 leader_election.cc:290] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.322559 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.322692 1048 raft_consensus.cc:3058] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.323014 2217 raft_consensus.cc:2747] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.323807 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f4cf919a96f34167b7dc9c3dfe7946a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:54.324404 850 consensus_peers.cc:489] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 -> Peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Couldn't send request to peer ef5c5979bb9e475f983af7c3e9e0e019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7b8cf118eb4a5c997815b97b9b7d00. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:54.325098 1048 raft_consensus.cc:2466] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:54.325449 710 leader_election.cc:304] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:54.325798 2052 raft_consensus.cc:2802] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.326014 2052 raft_consensus.cc:695] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.326260 2052 consensus_queue.cc:237] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.329775 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cfb025535ecd4cf88681c4951a56ba29
I20250629 01:51:54.329938 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_57474f12b4c04c9c8a9e6e85838cdb8f" } new_table_name: "default.Table_4adf3f46833048789a2485d973c3bf59" modify_external_catalogs: true
I20250629 01:51:54.331081 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet cfb025535ecd4cf88681c4951a56ba29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:54.329777 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 45461dc364d847d0b0272ac714559008 consensus state has no leader
I20250629 01:51:54.331338 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cfb025535ecd4cf88681c4951a56ba29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.331012 1157 ts_tablet_manager.cc:1918] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.332139 1157 log.cc:1199] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0b16b6840c24484b90218fffc1da9621
I20250629 01:51:54.333699 881 tablet_service.cc:1515] Processing DeleteTablet for tablet cfb025535ecd4cf88681c4951a56ba29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:54.335314 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c46bd4bfa331420abdcd5fe729300fc4 consensus state has no leader
I20250629 01:51:54.335191 1157 ts_tablet_manager.cc:1939] T 0b16b6840c24484b90218fffc1da9621 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.338265 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 869) took 1636 ms. Trace:
I20250629 01:51:54.339540 1155 tablet_bootstrap.cc:492] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.339692 1155 ts_tablet_manager.cc:1397] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250629 01:51:54.338933 1157 rpcz_store.cc:276] 0629 01:51:52.701416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.701466 (+ 50us) service_pool.cc:225] Handling call
0629 01:51:54.283902 (+1582436us) ts_tablet_manager.cc:1145] Deleting tablet 0b16b6840c24484b90218fffc1da9621
0629 01:51:54.283909 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.307684 (+ 23775us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.330366 (+ 22682us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.338035 (+ 7669us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582298}
I20250629 01:51:54.340358 1155 raft_consensus.cc:357] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.340647 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0b16b6840c24484b90218fffc1da9621 (table default.Table_10a0ae1fcff049dda2c11881c3191fa3 [id=14974743edd741fbab51f9e8a4add096]) successfully deleted
I20250629 01:51:54.340902 1155 raft_consensus.cc:383] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.341005 1155 raft_consensus.cc:738] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.341238 1155 consensus_queue.cc:260] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.341508 1157 tablet_replica.cc:331] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.341807 1155 ts_tablet_manager.cc:1428] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:54.342094 1155 tablet_bootstrap.cc:492] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.343462 1157 raft_consensus.cc:2241] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.278450 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 937) took 1056 ms. Trace:
I20250629 01:51:54.344633 1156 rpcz_store.cc:276] 0629 01:51:53.221848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.221899 (+ 51us) service_pool.cc:225] Handling call
0629 01:51:54.264320 (+1042421us) ts_tablet_manager.cc:1145] Deleting tablet 90eef69522e54b12b849d94e56355921
0629 01:51:54.278351 (+ 14031us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.278434 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042269}
I20250629 01:51:54.345078 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 938) took 1118 ms. Trace:
I20250629 01:51:54.344712 1157 raft_consensus.cc:2270] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:54.345117 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 90eef69522e54b12b849d94e56355921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90eef69522e54b12b849d94e56355921 already in progress: creating tablet
I20250629 01:51:54.345572 1156 rpcz_store.cc:276] 0629 01:51:53.226359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.226405 (+ 46us) service_pool.cc:225] Handling call
0629 01:51:54.344970 (+1118565us) ts_tablet_manager.cc:1145] Deleting tablet e304f0bfe13846f29e6390d9904a4f57
0629 01:51:54.344978 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.345049 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118429}
I20250629 01:51:54.346413 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 940) took 1061 ms. Trace:
W20250629 01:51:54.346469 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e304f0bfe13846f29e6390d9904a4f57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e304f0bfe13846f29e6390d9904a4f57 already in progress: creating tablet
I20250629 01:51:54.347036 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:54.347064 1156 rpcz_store.cc:276] 0629 01:51:53.285056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.285098 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:54.346314 (+1061216us) ts_tablet_manager.cc:1145] Deleting tablet 1e4a173d9e70453c91d8ff79a710bc00
0629 01:51:54.346322 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.346385 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1061088}
I20250629 01:51:54.347389 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 939) took 1062 ms. Trace:
W20250629 01:51:54.347461 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1e4a173d9e70453c91d8ff79a710bc00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e4a173d9e70453c91d8ff79a710bc00 already in progress: creating tablet
I20250629 01:51:54.347481 1156 rpcz_store.cc:276] 0629 01:51:53.284765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.287156 (+ 2391us) service_pool.cc:225] Handling call
0629 01:51:54.347293 (+1060137us) ts_tablet_manager.cc:1145] Deleting tablet 0f05d54fc9e649faabee85a418ebf2bb
0629 01:51:54.347302 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.347362 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1059963}
I20250629 01:51:54.347777 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 941) took 1018 ms. Trace:
W20250629 01:51:54.347862 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0f05d54fc9e649faabee85a418ebf2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f05d54fc9e649faabee85a418ebf2bb already in progress: creating tablet
I20250629 01:51:54.348960 1157 ts_tablet_manager.cc:1905] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.351925 1861 raft_consensus.cc:491] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.352053 1861 raft_consensus.cc:513] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.352525 1861 leader_election.cc:290] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.352887 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.353060 1049 raft_consensus.cc:2466] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.352994 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.353451 710 leader_election.cc:304] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.353729 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d1eb7cb229e741969597fce7379b7d40" } modify_external_catalogs: true
I20250629 01:51:54.353802 1861 raft_consensus.cc:2802] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.353883 1861 raft_consensus.cc:491] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.353984 1861 raft_consensus.cc:3058] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.356122 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d1eb7cb229e741969597fce7379b7d40" } modify_external_catalogs: true
W20250629 01:51:54.356292 712 leader_election.cc:343] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.357120 1861 raft_consensus.cc:513] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.358090 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.358373 1049 raft_consensus.cc:3058] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.357864 1158 ts_tablet_manager.cc:1918] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.358405 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:54.358736 712 leader_election.cc:343] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.358793 1158 log.cc:1199] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e3b17172ab8b4e0dad36ebaac776dc03
I20250629 01:51:54.361008 1049 raft_consensus.cc:2466] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:54.361483 710 leader_election.cc:304] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:54.361346 1158 ts_tablet_manager.cc:1939] T e3b17172ab8b4e0dad36ebaac776dc03 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.361860 1155 tablet_bootstrap.cc:654] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.362445 1861 leader_election.cc:290] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.362866 1861 raft_consensus.cc:2802] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.363204 1861 raft_consensus.cc:695] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.363622 1861 consensus_queue.cc:237] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.364195 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e3b17172ab8b4e0dad36ebaac776dc03 (table default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]) successfully deleted
W20250629 01:51:54.366892 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f4cf919a96f34167b7dc9c3dfe7946a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4cf919a96f34167b7dc9c3dfe7946a4 already in progress: creating tablet
I20250629 01:51:54.367311 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 54aa32f395ad47f39a97243594aa99e9
W20250629 01:51:54.368513 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet cfb025535ecd4cf88681c4951a56ba29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfb025535ecd4cf88681c4951a56ba29 already in progress: creating tablet
I20250629 01:51:54.370048 1155 tablet_bootstrap.cc:492] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.371205 1155 ts_tablet_manager.cc:1397] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250629 01:51:54.371800 1156 rpcz_store.cc:276] 0629 01:51:53.343182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.343307 (+ 125us) service_pool.cc:225] Handling call
0629 01:51:54.347676 (+1004369us) ts_tablet_manager.cc:1145] Deleting tablet 19b642dd58e948329442964df736b4a4
0629 01:51:54.347684 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.347748 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1004229}
I20250629 01:51:54.372028 1155 raft_consensus.cc:357] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:54.372326 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 19b642dd58e948329442964df736b4a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b642dd58e948329442964df736b4a4 already in progress: creating tablet
I20250629 01:51:54.373991 1155 raft_consensus.cc:383] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.373699 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bbbeb43473a4f6f892056a30f62589f with delete_type TABLET_DATA_DELETED (replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.374171 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e354fce951943d69e4c666897f5caac with delete_type TABLET_DATA_DELETED (replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.372686 1156 tablet_replica.cc:331] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.374459 1156 raft_consensus.cc:2241] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.374662 1156 raft_consensus.cc:2270] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.375180 1156 ts_tablet_manager.cc:1905] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.380153 1155 raft_consensus.cc:738] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.383846 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3812c4a8d39844e69e8e142dbf156fae" } modify_external_catalogs: true
I20250629 01:51:54.380044 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4e7128f0007c44f691cd8bc0931dfe17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.384225 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
W20250629 01:51:54.384374 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e7128f0007c44f691cd8bc0931dfe17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.387780 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3812c4a8d39844e69e8e142dbf156fae" } modify_external_catalogs: true
I20250629 01:51:54.394656 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 798516e55ca747fcb2942026adb78af0
I20250629 01:51:54.396436 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_04bf790258f64e6f87859f34119d5c4d" } modify_external_catalogs: true
I20250629 01:51:54.396575 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_04bf790258f64e6f87859f34119d5c4d" } modify_external_catalogs: true
I20250629 01:51:54.397241 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 798516e55ca747fcb2942026adb78af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.397457 1158 tablet_replica.cc:331] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.397691 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 798516e55ca747fcb2942026adb78af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.398018 1158 raft_consensus.cc:2241] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.398203 1159 tablet_bootstrap.cc:492] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.398274 1158 raft_consensus.cc:2270] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.398934 1158 ts_tablet_manager.cc:1905] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.399417 1154 tablet_bootstrap.cc:492] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.399511 1154 ts_tablet_manager.cc:1397] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.136s user 0.006s sys 0.014s
I20250629 01:51:54.400458 1159 ts_tablet_manager.cc:1397] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.148s user 0.014s sys 0.007s
I20250629 01:51:54.400687 2119 tablet.cc:1722] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:54.401007 1159 raft_consensus.cc:357] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.401144 1159 raft_consensus.cc:383] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.401186 1159 raft_consensus.cc:738] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.406152 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0dff81ebac6447489df5d50eea2401ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.406477 1154 raft_consensus.cc:357] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.406617 1154 raft_consensus.cc:383] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.406658 1154 raft_consensus.cc:738] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:54.406623 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dff81ebac6447489df5d50eea2401ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.406842 1159 consensus_queue.cc:260] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.412657 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5377f24c99e3434a805be34e0bcb9a39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.413087 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5377f24c99e3434a805be34e0bcb9a39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.409735 916 raft_consensus.cc:3058] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.415711 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0891a717f4824360be1c4fe4e773a511" } new_table_name: "default.Table_dc3b011b71e3400bafeb102d3c3cd403" modify_external_catalogs: true
I20250629 01:51:54.416268 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8a9b6060d8cd47aa8e1019d961b2fbab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.416666 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a9b6060d8cd47aa8e1019d961b2fbab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.417762 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_69f7afd6dbc24708810b9a825012ae62" } new_table_name: "default.Table_4fc8585bd5184ffeb9272ac437db94a8" modify_external_catalogs: true
I20250629 01:51:54.417898 1157 ts_tablet_manager.cc:1918] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.418012 1157 log.cc:1199] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/543242e4a5064268ae1e5519b569f682
I20250629 01:51:54.418357 1157 ts_tablet_manager.cc:1939] T 543242e4a5064268ae1e5519b569f682 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.418684 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 870) took 1698 ms. Trace:
I20250629 01:51:54.418759 1157 rpcz_store.cc:276] 0629 01:51:52.719698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.719737 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:54.341123 (+1621386us) ts_tablet_manager.cc:1145] Deleting tablet 543242e4a5064268ae1e5519b569f682
0629 01:51:54.341132 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.373164 (+ 32032us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.417799 (+ 44635us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.418652 (+ 853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620964}
I20250629 01:51:54.419169 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 543242e4a5064268ae1e5519b569f682 (table default.Table_1048b0cf35964b1fb36e7004e009ac0e [id=05fdb8424b714d79a1e7077c3d502763]) successfully deleted
I20250629 01:51:54.419529 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 871) took 1692 ms. Trace:
I20250629 01:51:54.419634 1157 rpcz_store.cc:276] 0629 01:51:52.727109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.727151 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:54.419391 (+1692240us) ts_tablet_manager.cc:1145] Deleting tablet 4cc7e839241e447db039b261421a7fef
0629 01:51:54.419399 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.419515 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1692107}
W20250629 01:51:54.420030 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4cc7e839241e447db039b261421a7fef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cc7e839241e447db039b261421a7fef already in progress: creating tablet
I20250629 01:51:54.420293 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_efd88358bbde43528c5d3013bb7aaaf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.420416 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 872) took 1671 ms. Trace:
W20250629 01:51:54.420665 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efd88358bbde43528c5d3013bb7aaaf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.420524 1157 rpcz_store.cc:276] 0629 01:51:52.748867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.748909 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:54.420276 (+1671367us) ts_tablet_manager.cc:1145] Deleting tablet 59cc1690878f40e39e23e1942c75effa
0629 01:51:54.420285 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.420402 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1671226}
W20250629 01:51:54.421005 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 59cc1690878f40e39e23e1942c75effa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59cc1690878f40e39e23e1942c75effa already in progress: creating tablet
I20250629 01:51:54.422290 1157 tablet_replica.cc:331] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.422587 1157 raft_consensus.cc:2241] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.422808 1157 raft_consensus.cc:2270] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.423166 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dae51ebf34aa422a9e59b9369d5564a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.423555 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dae51ebf34aa422a9e59b9369d5564a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.427569 1154 consensus_queue.cc:260] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.428131 1154 ts_tablet_manager.cc:1428] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250629 01:51:54.428418 1154 tablet_bootstrap.cc:492] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.420419 1159 ts_tablet_manager.cc:1428] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250629 01:51:54.433478 1159 tablet_bootstrap.cc:492] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.435163 1159 tablet_bootstrap.cc:654] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.439736 1157 ts_tablet_manager.cc:1905] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.441813 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
I20250629 01:51:54.442412 1158 ts_tablet_manager.cc:1918] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:54.442340 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 9cf553b291854a6f9e7ed78b5db5f18c from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.443284 1158 log.cc:1199] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/798516e55ca747fcb2942026adb78af0
I20250629 01:51:54.446213 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 432680cae61946af9859760fe23110ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.453589 1158 ts_tablet_manager.cc:1939] T 798516e55ca747fcb2942026adb78af0 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.454699 2217 raft_consensus.cc:491] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.454805 2217 raft_consensus.cc:513] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.455253 2217 leader_election.cc:290] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:54.456048 850 leader_election.cc:343] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:54.456076 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_39a0ff1dd8064dc4a2990b95fac35099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.456465 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39a0ff1dd8064dc4a2990b95fac35099 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:54.457547 850 leader_election.cc:343] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:54.457653 850 leader_election.cc:304] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.457973 2217 raft_consensus.cc:2747] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.458655 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]) successfully deleted
W20250629 01:51:54.460731 440 catalog_manager.cc:5001] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): alter failed for tablet cf4b52fae47f45ad9d77b0a97a854860 (table default.Table_72800b006f6e4b2eab6c57ecea71aef0 [id=53d17bca03a74467a681cb07cb85ef23]),no further retry: Not found: Tablet not found: cf4b52fae47f45ad9d77b0a97a854860
I20250629 01:51:54.463933 2233 tablet.cc:1722] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:54.464473 1159 tablet_bootstrap.cc:492] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.464563 1159 ts_tablet_manager.cc:1397] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.001s
I20250629 01:51:54.465046 1159 raft_consensus.cc:357] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.465158 1159 raft_consensus.cc:383] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.465193 1159 raft_consensus.cc:738] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.465404 1159 consensus_queue.cc:260] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.467409 1159 ts_tablet_manager.cc:1428] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:54.468274 1159 tablet_bootstrap.cc:492] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.395869 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 798516e55ca747fcb2942026adb78af0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.451398 2238 tablet.cc:1722] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:54.470021 1155 consensus_queue.cc:260] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:54.470602 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b11d32f1c3e41f3b9f6d20fd837226a consensus state has no leader
I20250629 01:51:54.471155 1155 ts_tablet_manager.cc:1428] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.100s user 0.001s sys 0.000s
I20250629 01:51:54.472810 1861 raft_consensus.cc:491] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.474164 1861 raft_consensus.cc:513] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.474265 1155 tablet_bootstrap.cc:492] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.474681 1861 leader_election.cc:290] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.475208 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6d79ac3c5f04b5db295c93c3aae92cb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.475651 712 leader_election.cc:343] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.477128 1155 tablet_bootstrap.cc:654] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:54.477165 710 leader_election.cc:343] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f6d79ac3c5f04b5db295c93c3aae92cb
I20250629 01:51:54.477276 710 leader_election.cc:304] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.477630 2228 raft_consensus.cc:491] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.477730 2228 raft_consensus.cc:513] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.478250 2228 leader_election.cc:290] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.478286 1156 ts_tablet_manager.cc:1918] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.478399 2228 raft_consensus.cc:2747] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.480435 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8b38c20eb1849b1a9f8acd40315eef5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.478396 1156 log.cc:1199] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8dd084685cc0407da9e12b0a4df386e9
I20250629 01:51:54.481678 1156 ts_tablet_manager.cc:1939] T 8dd084685cc0407da9e12b0a4df386e9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:54.482172 712 leader_election.cc:343] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:54.483073 710 leader_election.cc:343] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:54.483217 710 leader_election.cc:304] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.483577 1861 raft_consensus.cc:2747] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.484367 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 943) took 1115 ms. Trace:
I20250629 01:51:54.484485 1156 rpcz_store.cc:276] 0629 01:51:53.369129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.369172 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:54.372600 (+1003428us) ts_tablet_manager.cc:1145] Deleting tablet 8dd084685cc0407da9e12b0a4df386e9
0629 01:51:54.372611 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.378029 (+ 5418us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.478180 (+100151us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.484310 (+ 6130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003298}
I20250629 01:51:54.484875 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8dd084685cc0407da9e12b0a4df386e9 (table default.Table_875984d711bd4f7da73063fdc6fba264 [id=af75db4dc824485997f324795f65607b]) successfully deleted
I20250629 01:51:54.485210 1156 tablet_replica.cc:331] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.485401 1156 raft_consensus.cc:2241] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.485579 1156 raft_consensus.cc:2270] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.486029 1156 ts_tablet_manager.cc:1905] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.490127 1159 tablet_bootstrap.cc:654] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.494239 1155 tablet_bootstrap.cc:492] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.494393 1155 ts_tablet_manager.cc:1397] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250629 01:51:54.495015 1155 raft_consensus.cc:357] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.495198 1155 raft_consensus.cc:383] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.495303 1155 raft_consensus.cc:738] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.495533 1155 consensus_queue.cc:260] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.495997 1155 ts_tablet_manager.cc:1428] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:54.497996 1155 tablet_bootstrap.cc:492] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.499465 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a0699bfad44f4661b365b0924fd37513"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.499886 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0699bfad44f4661b365b0924fd37513 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.500001 1154 tablet_bootstrap.cc:654] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.500021 1155 tablet_bootstrap.cc:654] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.501579 1048 raft_consensus.cc:1273] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:54.501837 712 consensus_peers.cc:489] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:54.502274 1156 ts_tablet_manager.cc:1918] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.502403 1156 log.cc:1199] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/897a4cefefe04f06b47070ec7fa1a33b
I20250629 01:51:54.502725 1156 ts_tablet_manager.cc:1939] T 897a4cefefe04f06b47070ec7fa1a33b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.503397 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 945) took 1058 ms. Trace:
I20250629 01:51:54.503583 1861 consensus_queue.cc:1035] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:54.503541 1156 rpcz_store.cc:276] 0629 01:51:53.444595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.447616 (+ 3021us) service_pool.cc:225] Handling call
0629 01:51:54.485115 (+1037499us) ts_tablet_manager.cc:1145] Deleting tablet 897a4cefefe04f06b47070ec7fa1a33b
0629 01:51:54.485125 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.496327 (+ 11202us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.502173 (+ 5846us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.503363 (+ 1190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1037268}
I20250629 01:51:54.504015 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 897a4cefefe04f06b47070ec7fa1a33b (table default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]) successfully deleted
I20250629 01:51:54.504338 1156 tablet_replica.cc:331] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.504725 1156 raft_consensus.cc:2241] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.505291 1156 raft_consensus.cc:2270] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.508949 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:54.510846 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50ab66cf5f2048fcb80e323c0935d83f consensus state has no leader
I20250629 01:51:54.512143 1155 tablet_bootstrap.cc:492] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.512279 1155 ts_tablet_manager.cc:1397] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250629 01:51:54.512821 1155 raft_consensus.cc:357] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.512979 1155 raft_consensus.cc:383] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.513041 1155 raft_consensus.cc:738] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.513226 1155 consensus_queue.cc:260] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.513605 1155 ts_tablet_manager.cc:1428] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:54.513939 1155 tablet_bootstrap.cc:492] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.515724 1155 tablet_bootstrap.cc:654] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.521099 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3699a1e4f7ec403ba943711d0c3e6588" } new_table_name: "default.Table_80d068bf778042d1b60dfbad2b5e36a7" modify_external_catalogs: true
I20250629 01:51:54.521996 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_b55682d2903c476d9754ce8cc842594c [id=80262dd3324a4c2390da9adb2540dd62] alter complete (version 1)
I20250629 01:51:54.522110 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c74c8328d4ae482196d5c984598ad171" } modify_external_catalogs: true
I20250629 01:51:54.522213 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c74c8328d4ae482196d5c984598ad171" } modify_external_catalogs: true
I20250629 01:51:54.522507 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 34172d05ae40417494cd8f67f4cb981e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:42 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.530159 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d5042d6281a54e70861a5619a2967981"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.530766 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5042d6281a54e70861a5619a2967981 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.531106 2238 tablet.cc:1722] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:54.532588 881 tablet_service.cc:1468] Processing CreateTablet for tablet 73908c5ee60f4a838216733b7ce2b5c4 (DEFAULT_TABLE table=default.Table_cbc70b839d4f4fd89897b14344c2b037 [id=7cd6e89651fc4a898c6015d481236ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.532919 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73908c5ee60f4a838216733b7ce2b5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.533008 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 73908c5ee60f4a838216733b7ce2b5c4 (DEFAULT_TABLE table=default.Table_cbc70b839d4f4fd89897b14344c2b037 [id=7cd6e89651fc4a898c6015d481236ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.533306 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73908c5ee60f4a838216733b7ce2b5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.531044 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:54.533529 446 catalog_manager.cc:5582] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:54.533803 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]). Expected version 2 got 0
I20250629 01:51:54.533975 446 catalog_manager.cc:5582] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:54.534200 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 0323947b135a4308903b1eed1ca4db71 (table default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]). Expected version 3 got 0
I20250629 01:51:54.534356 446 catalog_manager.cc:5582] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:54.534586 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet ddc24d0e668c49ac825b2578849ab2a3 (table default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e]). Expected version 2 got 0
I20250629 01:51:54.534754 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_f798e82e14704a689fe84cbeb9a8d477 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 4 got 0
I20250629 01:51:54.535627 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bbfbc9cc8c0f441c8718c1ea68f3a287" } new_table_name: "default.Table_173755c0c4634d9484470c948516f837" modify_external_catalogs: true
I20250629 01:51:54.531339 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
W20250629 01:51:54.533306 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5d2f2975e5d64efd888ed948f1420916 consensus state has no leader
I20250629 01:51:54.536527 2049 raft_consensus.cc:491] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.536638 2049 raft_consensus.cc:513] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.537065 2049 leader_election.cc:290] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.537145 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20d462c5b8284f99b445c7539d756d3b" } new_table_name: "default.Table_dc8c2079eef0455a8ba102bd39414e15" modify_external_catalogs: true
I20250629 01:51:54.538225 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb4ff3220cf4d25b680235f1b4882ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.538358 1050 raft_consensus.cc:2466] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.538790 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb4ff3220cf4d25b680235f1b4882ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.539198 712 leader_election.cc:343] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.539562 710 leader_election.cc:304] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:54.539919 2049 raft_consensus.cc:2802] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.539980 2049 raft_consensus.cc:491] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.540031 2049 raft_consensus.cc:3058] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.540390 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6f23d8816c084fafa5c57899912ffdd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.540756 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f23d8816c084fafa5c57899912ffdd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.541574 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a7f78eef06dd413cad0a63d1923e1263" } modify_external_catalogs: true
I20250629 01:51:54.541685 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a7f78eef06dd413cad0a63d1923e1263" } modify_external_catalogs: true
I20250629 01:51:54.542199 2049 raft_consensus.cc:513] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.542935 2233 tablet.cc:1722] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:54.543308 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]). Expected version 1 got 0
I20250629 01:51:54.543583 1157 ts_tablet_manager.cc:1918] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.543674 1157 log.cc:1199] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/bd5150eff1ad43588a9e70dab47744e3
I20250629 01:51:54.543933 1157 ts_tablet_manager.cc:1939] T bd5150eff1ad43588a9e70dab47744e3 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.544193 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 877) took 1754 ms. Trace:
I20250629 01:51:54.544261 1157 rpcz_store.cc:276] 0629 01:51:52.790121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.805860 (+ 15739us) service_pool.cc:225] Handling call
0629 01:51:54.422228 (+1616368us) ts_tablet_manager.cc:1145] Deleting tablet bd5150eff1ad43588a9e70dab47744e3
0629 01:51:54.422236 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.489856 (+ 67620us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.543498 (+ 53642us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.544178 (+ 680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1616207}
I20250629 01:51:54.544562 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 878) took 1731 ms. Trace:
I20250629 01:51:54.544620 1157 rpcz_store.cc:276] 0629 01:51:52.813514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.818773 (+ 5259us) service_pool.cc:225] Handling call
0629 01:51:54.544475 (+1725702us) ts_tablet_manager.cc:1145] Deleting tablet 62369ffc1d4a4628bb24254877d6dee6
0629 01:51:54.544483 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.544549 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725528}
I20250629 01:51:54.544869 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 879) took 1725 ms. Trace:
I20250629 01:51:54.544922 1157 rpcz_store.cc:276] 0629 01:51:52.819004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.826520 (+ 7516us) service_pool.cc:225] Handling call
0629 01:51:54.544786 (+1718266us) ts_tablet_manager.cc:1145] Deleting tablet ec3937679b37476f934aceb30ac62d68
0629 01:51:54.544792 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.544850 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1718114}
I20250629 01:51:54.545133 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 883) took 1674 ms. Trace:
I20250629 01:51:54.545181 1157 rpcz_store.cc:276] 0629 01:51:52.870712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.885907 (+ 15195us) service_pool.cc:225] Handling call
0629 01:51:54.545067 (+1659160us) ts_tablet_manager.cc:1145] Deleting tablet 730178bc660f4af596007721df96198f
0629 01:51:54.545073 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.545123 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1659028}
I20250629 01:51:54.545414 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 885) took 1641 ms. Trace:
I20250629 01:51:54.545471 1157 rpcz_store.cc:276] 0629 01:51:52.904250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.910114 (+ 5864us) service_pool.cc:225] Handling call
0629 01:51:54.545334 (+1635220us) ts_tablet_manager.cc:1145] Deleting tablet 64d3efa5b59c4ae6af52c7490c316d7e
0629 01:51:54.545342 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.545400 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1635075}
I20250629 01:51:54.545710 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 887) took 1637 ms. Trace:
I20250629 01:51:54.545765 1157 rpcz_store.cc:276] 0629 01:51:52.908135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.916718 (+ 8583us) service_pool.cc:225] Handling call
0629 01:51:54.545629 (+1628911us) ts_tablet_manager.cc:1145] Deleting tablet e841a0c33e504a5081bae39d731106cb
0629 01:51:54.545637 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.545698 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628774}
I20250629 01:51:54.545958 1157 tablet_replica.cc:331] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.546103 1157 raft_consensus.cc:2241] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.546213 1157 raft_consensus.cc:2270] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.546633 1157 ts_tablet_manager.cc:1905] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.546949 2052 raft_consensus.cc:491] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.547046 2052 raft_consensus.cc:513] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.547477 2052 leader_election.cc:290] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.547812 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb4ff3220cf4d25b680235f1b4882ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:54.548499 712 leader_election.cc:343] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.548612 2049 leader_election.cc:290] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.548894 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5c6f3f19a4d441f58c8c6099e04f82ec" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.549254 712 leader_election.cc:343] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.549543 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet bd5150eff1ad43588a9e70dab47744e3 (table default.Table_99e79a9aa55444b6b6d1873d0d4844c8 [id=c483797822d647458b5728bf3255b15d]) successfully deleted
W20250629 01:51:54.549701 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 62369ffc1d4a4628bb24254877d6dee6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62369ffc1d4a4628bb24254877d6dee6 already in progress: creating tablet
W20250629 01:51:54.549903 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 730178bc660f4af596007721df96198f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 730178bc660f4af596007721df96198f already in progress: creating tablet
W20250629 01:51:54.550025 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 64d3efa5b59c4ae6af52c7490c316d7e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64d3efa5b59c4ae6af52c7490c316d7e already in progress: creating tablet
W20250629 01:51:54.550181 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e841a0c33e504a5081bae39d731106cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e841a0c33e504a5081bae39d731106cb already in progress: creating tablet
I20250629 01:51:54.551028 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bb4ff3220cf4d25b680235f1b4882ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.551162 1050 raft_consensus.cc:3058] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.553373 1050 raft_consensus.cc:2466] T 1bb4ff3220cf4d25b680235f1b4882ed P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:51:54.553860 710 leader_election.cc:343] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 5c6f3f19a4d441f58c8c6099e04f82ec
I20250629 01:51:54.553941 710 leader_election.cc:304] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.554289 710 leader_election.cc:304] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:54.554566 2228 raft_consensus.cc:2747] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.554740 2228 raft_consensus.cc:2802] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.554814 2228 raft_consensus.cc:695] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.554998 2228 consensus_queue.cc:237] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.562223 1159 tablet_bootstrap.cc:492] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.562595 1159 ts_tablet_manager.cc:1397] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.001s
I20250629 01:51:54.563545 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 81b32bac5a2144d1943285f5f446d53d from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.564028 1159 raft_consensus.cc:357] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.564203 1159 raft_consensus.cc:383] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.564258 1159 raft_consensus.cc:738] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.564437 1159 consensus_queue.cc:260] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.566777 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 109e19ed0ea64fa4985e93efbaca5107
I20250629 01:51:54.567638 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 109e19ed0ea64fa4985e93efbaca5107 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.569044 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 109e19ed0ea64fa4985e93efbaca5107 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.570159 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_db86a323f87542f1ac0296e67733fa1d" } modify_external_catalogs: true
I20250629 01:51:54.570191 888 tablet_service.cc:1515] Processing DeleteTablet for tablet a485e0e6e2f04aa6bd67a62f6d30be26 with delete_type TABLET_DATA_DELETED (replaced by tablet 462789e8daef42a08ff8b2cd2ddd1c49) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.570276 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_db86a323f87542f1ac0296e67733fa1d" } modify_external_catalogs: true
I20250629 01:51:54.571172 1159 ts_tablet_manager.cc:1428] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
W20250629 01:51:54.571390 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 109e19ed0ea64fa4985e93efbaca5107 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 109e19ed0ea64fa4985e93efbaca5107 already in progress: creating tablet
I20250629 01:51:54.572721 1159 tablet_bootstrap.cc:492] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.574393 1159 tablet_bootstrap.cc:654] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.574429 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 109e19ed0ea64fa4985e93efbaca5107 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:54.584147 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c83cc13930f4c99b9bc23603eef220f consensus state has no leader
W20250629 01:51:54.584497 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 290c5ab8364145a39f6ff403ab7b20b1 consensus state has no leader
I20250629 01:51:54.585006 1154 tablet_bootstrap.cc:492] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.585167 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8992da4b777d400b9c3f242444229353
I20250629 01:51:54.585186 1154 ts_tablet_manager.cc:1397] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.013s
I20250629 01:51:54.585942 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8992da4b777d400b9c3f242444229353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.586151 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 8992da4b777d400b9c3f242444229353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.586174 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 8992da4b777d400b9c3f242444229353 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.586383 1154 raft_consensus.cc:357] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.586676 1154 raft_consensus.cc:383] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.586786 1154 raft_consensus.cc:738] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:54.587090 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8992da4b777d400b9c3f242444229353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8992da4b777d400b9c3f242444229353 already in progress: creating tablet
I20250629 01:51:54.587246 1155 tablet_bootstrap.cc:492] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.587071 1154 consensus_queue.cc:260] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.587383 1155 ts_tablet_manager.cc:1397] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.001s
I20250629 01:51:54.587970 1155 raft_consensus.cc:357] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.588156 1155 raft_consensus.cc:383] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.588200 1155 raft_consensus.cc:738] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.588371 1155 consensus_queue.cc:260] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.588809 1155 ts_tablet_manager.cc:1428] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:54.589030 1155 tablet_bootstrap.cc:492] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.589296 881 tablet_service.cc:1468] Processing CreateTablet for tablet 189c6c3627404c2685d77cb3b12cace8 (DEFAULT_TABLE table=default.Table_1b3364a7417244a59e0dc1cae5101612 [id=8b58e10a65bf4077827712d0f8ded72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.589581 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189c6c3627404c2685d77cb3b12cace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.590183 749 tablet_service.cc:1468] Processing CreateTablet for tablet 189c6c3627404c2685d77cb3b12cace8 (DEFAULT_TABLE table=default.Table_1b3364a7417244a59e0dc1cae5101612 [id=8b58e10a65bf4077827712d0f8ded72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.590446 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189c6c3627404c2685d77cb3b12cace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.590533 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 189c6c3627404c2685d77cb3b12cace8 (DEFAULT_TABLE table=default.Table_1b3364a7417244a59e0dc1cae5101612 [id=8b58e10a65bf4077827712d0f8ded72c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.590807 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 189c6c3627404c2685d77cb3b12cace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.591914 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5614713dab134cb9b6d70c9a53973014
I20250629 01:51:54.587985 743 tablet_service.cc:1468] Processing CreateTablet for tablet 73908c5ee60f4a838216733b7ce2b5c4 (DEFAULT_TABLE table=default.Table_cbc70b839d4f4fd89897b14344c2b037 [id=7cd6e89651fc4a898c6015d481236ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.592646 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73908c5ee60f4a838216733b7ce2b5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.595501 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 5614713dab134cb9b6d70c9a53973014 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.595865 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5614713dab134cb9b6d70c9a53973014 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.596450 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 5614713dab134cb9b6d70c9a53973014 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:54.596464 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5614713dab134cb9b6d70c9a53973014 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5614713dab134cb9b6d70c9a53973014 already in progress: creating tablet
I20250629 01:51:54.596881 1159 tablet_bootstrap.cc:492] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.597002 1159 ts_tablet_manager.cc:1397] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250629 01:51:54.597656 1159 raft_consensus.cc:357] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.597805 1159 raft_consensus.cc:383] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.597829 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92114ac6ae69479e86909ca0e6744dec" } modify_external_catalogs: true
I20250629 01:51:54.597884 1159 raft_consensus.cc:738] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.597962 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_92114ac6ae69479e86909ca0e6744dec" } modify_external_catalogs: true
I20250629 01:51:54.598130 1159 consensus_queue.cc:260] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.600281 888 tablet_service.cc:1468] Processing CreateTablet for tablet bbb61a515a4744c18d21d1b96f012f7b (DEFAULT_TABLE table=default.Table_730ee6fd4aca4315aaf72504cecdbad5 [id=6d428136d3e4453ba898fba94a408232]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.600556 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb61a515a4744c18d21d1b96f012f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.602181 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d925ffce89b40258b29ec021bfa5f92 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d21dfd48ea4a95a2fcecde6735974b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.602439 1159 ts_tablet_manager.cc:1428] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:54.602528 1154 ts_tablet_manager.cc:1428] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20250629 01:51:54.603116 1026 tablet_service.cc:1468] Processing CreateTablet for tablet bbb61a515a4744c18d21d1b96f012f7b (DEFAULT_TABLE table=default.Table_730ee6fd4aca4315aaf72504cecdbad5 [id=6d428136d3e4453ba898fba94a408232]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.603380 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb61a515a4744c18d21d1b96f012f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.603834 1159 tablet_bootstrap.cc:492] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.604198 454 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d6d00ba9eb8a40358b02d85c2352fcfd
I20250629 01:51:54.604483 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_551758fbd6ed485096040f978cfec8af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.604907 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_551758fbd6ed485096040f978cfec8af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.605929 1154 tablet_bootstrap.cc:492] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.609124 1159 tablet_bootstrap.cc:654] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.609977 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f] alter complete (version 1)
I20250629 01:51:54.614003 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 52be184d0480495a894442d6f72a0f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.614212 1158 tablet_replica.cc:331] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.614349 1158 raft_consensus.cc:2241] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.614459 1158 raft_consensus.cc:2270] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.614578 1154 tablet_bootstrap.cc:654] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.614969 1158 ts_tablet_manager.cc:1905] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.590214 1155 tablet_bootstrap.cc:654] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.612449 916 raft_consensus.cc:3058] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.619225 1156 ts_tablet_manager.cc:1905] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.625491 1158 ts_tablet_manager.cc:1918] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.625828 1158 log.cc:1199] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/52be184d0480495a894442d6f72a0f12
I20250629 01:51:54.626696 1158 ts_tablet_manager.cc:1939] T 52be184d0480495a894442d6f72a0f12 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.627969 1154 tablet_bootstrap.cc:492] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.628177 1154 ts_tablet_manager.cc:1397] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250629 01:51:54.628729 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 52be184d0480495a894442d6f72a0f12 (table default.Table_cbbc85d10dc546b59c06dd6296eaa2db [id=fadb6eabbead48b1a11795ffed0c4d81]) successfully deleted
I20250629 01:51:54.629145 1154 raft_consensus.cc:357] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.629384 1154 raft_consensus.cc:383] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.629474 1156 ts_tablet_manager.cc:1918] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.629503 1154 raft_consensus.cc:738] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.629568 1156 log.cc:1199] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/1ea681bd91e54e4898bcfb72bed6198f
I20250629 01:51:54.629834 1156 ts_tablet_manager.cc:1939] T 1ea681bd91e54e4898bcfb72bed6198f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.629834 1154 consensus_queue.cc:260] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.630085 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 946) took 1175 ms. Trace:
I20250629 01:51:54.630144 1156 rpcz_store.cc:276] 0629 01:51:53.454772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.454825 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:54.504284 (+1049459us) ts_tablet_manager.cc:1145] Deleting tablet 1ea681bd91e54e4898bcfb72bed6198f
0629 01:51:54.504292 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.625050 (+120758us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.629397 (+ 4347us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.630070 (+ 673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049249}
I20250629 01:51:54.630425 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 1ea681bd91e54e4898bcfb72bed6198f (table default.Table_1a2a249712224a0fa06b23867c043cfa [id=2fbd2c29b63c488ba11a1ed546f38da4]) successfully deleted
I20250629 01:51:54.630884 1155 tablet_bootstrap.cc:492] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.630995 1155 ts_tablet_manager.cc:1397] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.005s
I20250629 01:51:54.631592 1155 raft_consensus.cc:357] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.631736 1155 raft_consensus.cc:383] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.631781 1155 raft_consensus.cc:738] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.632066 1155 consensus_queue.cc:260] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.632288 1157 ts_tablet_manager.cc:1918] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.632362 1157 log.cc:1199] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e82b1352bb4d49769808dc402c85890f
I20250629 01:51:54.632565 1155 ts_tablet_manager.cc:1428] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:54.632814 1155 tablet_bootstrap.cc:492] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.634548 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.634752 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9e0d550773c949df9992be1f9f01b548
I20250629 01:51:54.634898 1157 ts_tablet_manager.cc:1939] T e82b1352bb4d49769808dc402c85890f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:54.634990 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a6a012ba891949538800581436ef5380 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6a012ba891949538800581436ef5380 already in progress: creating tablet
I20250629 01:51:54.635139 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 888) took 1726 ms. Trace:
I20250629 01:51:54.635200 1157 rpcz_store.cc:276] 0629 01:51:52.908235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.916901 (+ 8666us) service_pool.cc:225] Handling call
0629 01:51:54.545927 (+1629026us) ts_tablet_manager.cc:1145] Deleting tablet e82b1352bb4d49769808dc402c85890f
0629 01:51:54.545933 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.591570 (+ 45637us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.632224 (+ 40654us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.635124 (+ 2900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628898}
I20250629 01:51:54.635576 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e0d550773c949df9992be1f9f01b548 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.635591 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 886) took 1731 ms. Trace:
I20250629 01:51:54.635651 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e82b1352bb4d49769808dc402c85890f (table default.Table_14b2bea86460403f89586e3ebdbfd084 [id=ab297380d8bd4ffa8db5cfbff08519ec]) successfully deleted
I20250629 01:51:54.635650 1157 rpcz_store.cc:276] 0629 01:51:52.904354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.927184 (+ 22830us) service_pool.cc:225] Handling call
0629 01:51:54.635499 (+1708315us) ts_tablet_manager.cc:1145] Deleting tablet ac9d77381f7e4241aa546eaea3077168
0629 01:51:54.635506 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.635575 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1708152}
I20250629 01:51:54.635800 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e0d550773c949df9992be1f9f01b548 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.635915 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 897) took 1517 ms. Trace:
I20250629 01:51:54.635970 1157 rpcz_store.cc:276] 0629 01:51:53.118653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.125243 (+ 6590us) service_pool.cc:225] Handling call
0629 01:51:54.635823 (+1510580us) ts_tablet_manager.cc:1145] Deleting tablet bf110bfaab6948f3a32cdf45241b31a7
0629 01:51:54.635830 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.635902 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1503569}
I20250629 01:51:54.636143 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e0d550773c949df9992be1f9f01b548 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.636351 1157 tablet_replica.cc:331] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:54.636359 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ac9d77381f7e4241aa546eaea3077168 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9d77381f7e4241aa546eaea3077168 already in progress: creating tablet
W20250629 01:51:54.636619 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet bf110bfaab6948f3a32cdf45241b31a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf110bfaab6948f3a32cdf45241b31a7 already in progress: creating tablet
I20250629 01:51:54.637426 1155 tablet_bootstrap.cc:654] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.638244 1157 raft_consensus.cc:2241] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.638855 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 948) took 1177 ms. Trace:
I20250629 01:51:54.638931 1156 rpcz_store.cc:276] 0629 01:51:53.461734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.461771 (+ 37us) service_pool.cc:225] Handling call
0629 01:51:54.638751 (+1176980us) ts_tablet_manager.cc:1145] Deleting tablet a0769f77ea234dd7bccc2244c08b512b
0629 01:51:54.638762 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.638841 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1176850}
W20250629 01:51:54.639460 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a0769f77ea234dd7bccc2244c08b512b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0769f77ea234dd7bccc2244c08b512b already in progress: creating tablet
I20250629 01:51:54.639787 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 947) took 1180 ms. Trace:
I20250629 01:51:54.639881 1156 rpcz_store.cc:276] 0629 01:51:53.458915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.463294 (+ 4379us) service_pool.cc:225] Handling call
0629 01:51:54.639693 (+1176399us) ts_tablet_manager.cc:1145] Deleting tablet a0002dca4db24405afdce3e1076eb52e
0629 01:51:54.639700 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.639770 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1173407}
I20250629 01:51:54.639993 1154 ts_tablet_manager.cc:1428] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
W20250629 01:51:54.640283 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a0002dca4db24405afdce3e1076eb52e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0002dca4db24405afdce3e1076eb52e already in progress: creating tablet
I20250629 01:51:54.640578 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 950) took 1146 ms. Trace:
I20250629 01:51:54.640674 1156 rpcz_store.cc:276] 0629 01:51:53.493679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.493726 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:54.640491 (+1146765us) ts_tablet_manager.cc:1145] Deleting tablet 06e31520d3ad4c3395a59049755a4725
0629 01:51:54.640500 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.640564 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146632}
I20250629 01:51:54.640846 743 tablet_service.cc:1468] Processing CreateTablet for tablet bbb61a515a4744c18d21d1b96f012f7b (DEFAULT_TABLE table=default.Table_730ee6fd4aca4315aaf72504cecdbad5 [id=6d428136d3e4453ba898fba94a408232]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:54.641031 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 06e31520d3ad4c3395a59049755a4725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06e31520d3ad4c3395a59049755a4725 already in progress: creating tablet
I20250629 01:51:54.641137 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbb61a515a4744c18d21d1b96f012f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.641388 1157 raft_consensus.cc:2270] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.641481 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 951) took 1145 ms. Trace:
I20250629 01:51:54.641535 1156 rpcz_store.cc:276] 0629 01:51:53.495624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.495662 (+ 38us) service_pool.cc:225] Handling call
0629 01:51:54.641292 (+1145630us) ts_tablet_manager.cc:1145] Deleting tablet 07d5eb1b3c3345c099f298df240a5c75
0629 01:51:54.641299 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.641467 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1145513}
W20250629 01:51:54.641836 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 07d5eb1b3c3345c099f298df240a5c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07d5eb1b3c3345c099f298df240a5c75 already in progress: creating tablet
I20250629 01:51:54.642261 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 953) took 1106 ms. Trace:
I20250629 01:51:54.642546 1156 rpcz_store.cc:276] 0629 01:51:53.535985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.536033 (+ 48us) service_pool.cc:225] Handling call
0629 01:51:54.642185 (+1106152us) ts_tablet_manager.cc:1145] Deleting tablet 250ce9c0b27c40d996db138d0acb28d7
0629 01:51:54.642192 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.642248 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105979}
W20250629 01:51:54.642855 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 250ce9c0b27c40d996db138d0acb28d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250ce9c0b27c40d996db138d0acb28d7 already in progress: creating tablet
I20250629 01:51:54.639382 1159 tablet_bootstrap.cc:492] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.643558 1159 ts_tablet_manager.cc:1397] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
I20250629 01:51:54.643750 1154 tablet_bootstrap.cc:492] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.644639 1159 raft_consensus.cc:357] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.644778 1159 raft_consensus.cc:383] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.644827 1159 raft_consensus.cc:738] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.644999 1159 consensus_queue.cc:260] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.645375 1159 ts_tablet_manager.cc:1428] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:54.645582 1159 tablet_bootstrap.cc:492] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.645517 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_716fe511fb2c4fb384a34d9964136cab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.645959 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_716fe511fb2c4fb384a34d9964136cab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.646888 1159 tablet_bootstrap.cc:654] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.646850 2049 raft_consensus.cc:491] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:51:54.646940 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9e0d550773c949df9992be1f9f01b548 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e0d550773c949df9992be1f9f01b548 already in progress: creating tablet
I20250629 01:51:54.647059 2049 raft_consensus.cc:513] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:54.647146 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c already in progress: creating tablet
I20250629 01:51:54.648224 1156 tablet_replica.cc:331] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.648686 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef976703d61442f8987ff32070a33592" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.648713 1154 tablet_bootstrap.cc:654] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.649041 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_20c6268f43c64a5480574ad87d428a7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.649441 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20c6268f43c64a5480574ad87d428a7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.650086 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.650257 1158 tablet_replica.cc:331] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.650421 1158 raft_consensus.cc:2241] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.650566 1158 raft_consensus.cc:2270] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:54.649324 712 leader_election.cc:343] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:54.652117 710 leader_election.cc:343] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ef976703d61442f8987ff32070a33592
I20250629 01:51:54.652292 710 leader_election.cc:304] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.648031 2049 leader_election.cc:290] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.652599 2052 raft_consensus.cc:2747] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.652757 1156 raft_consensus.cc:2241] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.652972 1158 ts_tablet_manager.cc:1905] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.653219 1156 raft_consensus.cc:2270] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.654364 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_29b731f4ff984ce5bb98b903fbb804c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.654747 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29b731f4ff984ce5bb98b903fbb804c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.657142 1156 ts_tablet_manager.cc:1905] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.658385 2217 raft_consensus.cc:491] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.658481 2217 raft_consensus.cc:513] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.659029 2217 leader_election.cc:290] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:54.660229 850 leader_election.cc:343] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 942f7bbe29f94083b2494aab8fbcc0c5
W20250629 01:51:54.660741 850 leader_election.cc:343] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 942f7bbe29f94083b2494aab8fbcc0c5
I20250629 01:51:54.660822 850 leader_election.cc:304] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.661115 2217 raft_consensus.cc:2747] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.661527 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6e07ccb977954917826f8f245c6be17a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.661986 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e07ccb977954917826f8f245c6be17a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.668473 1157 ts_tablet_manager.cc:1905] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.668516 1156 ts_tablet_manager.cc:1918] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.668598 1156 log.cc:1199] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:54.670109 1156 ts_tablet_manager.cc:1939] T f8b38c20eb1849b1a9f8acd40315eef5 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.671134 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f8b38c20eb1849b1a9f8acd40315eef5 (table default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]) successfully deleted
W20250629 01:51:54.672125 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9cfff25bea764e62954e49611b71703c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cfff25bea764e62954e49611b71703c already in progress: creating tablet
W20250629 01:51:54.672840 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d1131e0d9c06435fb732a8a72e421298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1131e0d9c06435fb732a8a72e421298 already in progress: creating tablet
I20250629 01:51:54.673961 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3dd8aa3c2e914f019c0e7d29a3dbda13" } new_table_name: "default.Table_803f2f1324164400bb38186cbb242bcb" modify_external_catalogs: true
W20250629 01:51:54.673707 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet daf312e60c014fe698d3ec9f1a1b769c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf312e60c014fe698d3ec9f1a1b769c already in progress: creating tablet
W20250629 01:51:54.676285 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 74b3badd684d45fa88dc8cbcfc61001a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b3badd684d45fa88dc8cbcfc61001a already in progress: creating tablet
I20250629 01:51:54.676714 1156 tablet_replica.cc:331] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.677012 1156 raft_consensus.cc:2241] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.677305 1156 raft_consensus.cc:2270] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.678349 1156 ts_tablet_manager.cc:1905] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.682200 1158 ts_tablet_manager.cc:1918] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.686277 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 5d2f2975e5d64efd888ed948f1420916 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.687356 888 tablet_service.cc:1468] Processing CreateTablet for tablet c78405761d424300be42b6fe7900260e (DEFAULT_TABLE table=default.Table_8a23a625023340518638020e0ba2aab0 [id=a33c3321c9ae40909414679148a8a964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.687695 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c78405761d424300be42b6fe7900260e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.689450 881 tablet_service.cc:1468] Processing CreateTablet for tablet 18dd4a609fd4481896ad824e62b80df5 (DEFAULT_TABLE table=default.Table_0509ea2e580b4b818e39da38211d0a23 [id=21e87fe57a53452397ae36261cfd0833]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.689733 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dd4a609fd4481896ad824e62b80df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.693009 1158 log.cc:1199] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a6a012ba891949538800581436ef5380
I20250629 01:51:54.693023 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b296280dfbe1428b9e98ebbb01dacc44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.693248 1158 ts_tablet_manager.cc:1939] T a6a012ba891949538800581436ef5380 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:54.693393 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b296280dfbe1428b9e98ebbb01dacc44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.694435 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a1e8b8763bff46768cb4e15ba61d470b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.694813 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a6a012ba891949538800581436ef5380 (table default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]) successfully deleted
W20250629 01:51:54.694820 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1e8b8763bff46768cb4e15ba61d470b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.687592 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c78405761d424300be42b6fe7900260e (DEFAULT_TABLE table=default.Table_8a23a625023340518638020e0ba2aab0 [id=a33c3321c9ae40909414679148a8a964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.692193 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 18dd4a609fd4481896ad824e62b80df5 (DEFAULT_TABLE table=default.Table_0509ea2e580b4b818e39da38211d0a23 [id=21e87fe57a53452397ae36261cfd0833]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.695293 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dd4a609fd4481896ad824e62b80df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.697721 749 tablet_service.cc:1468] Processing CreateTablet for tablet 18dd4a609fd4481896ad824e62b80df5 (DEFAULT_TABLE table=default.Table_0509ea2e580b4b818e39da38211d0a23 [id=21e87fe57a53452397ae36261cfd0833]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.698004 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18dd4a609fd4481896ad824e62b80df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.699173 1157 ts_tablet_manager.cc:1918] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.699267 1157 log.cc:1199] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/86832a3c26224c7ea0b9b4b6866d3880
I20250629 01:51:54.699540 1157 ts_tablet_manager.cc:1939] T 86832a3c26224c7ea0b9b4b6866d3880 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.699795 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 898) took 1564 ms. Trace:
I20250629 01:51:54.699862 1157 rpcz_store.cc:276] 0629 01:51:53.135366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.152203 (+ 16837us) service_pool.cc:225] Handling call
0629 01:51:54.636314 (+1484111us) ts_tablet_manager.cc:1145] Deleting tablet 86832a3c26224c7ea0b9b4b6866d3880
0629 01:51:54.636321 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.672555 (+ 36234us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.699097 (+ 26542us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.699783 (+ 686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1483941}
I20250629 01:51:54.700268 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 86832a3c26224c7ea0b9b4b6866d3880 (table default.Table_f0a26d9ea3e8407da5ca0813db26beb3 [id=b52a4204ccf34819b3782aca3ea1db0f]) successfully deleted
I20250629 01:51:54.701169 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c78405761d424300be42b6fe7900260e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.702621 743 tablet_service.cc:1468] Processing CreateTablet for tablet c78405761d424300be42b6fe7900260e (DEFAULT_TABLE table=default.Table_8a23a625023340518638020e0ba2aab0 [id=a33c3321c9ae40909414679148a8a964]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.702906 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c78405761d424300be42b6fe7900260e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.703011 2052 raft_consensus.cc:491] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.703095 2052 raft_consensus.cc:513] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.703539 2052 leader_election.cc:290] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.704057 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a40c1a8eec9549a8bdf695d6516440a8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:54.704504 712 leader_election.cc:343] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:54.705080 710 leader_election.cc:343] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a40c1a8eec9549a8bdf695d6516440a8
I20250629 01:51:54.705178 710 leader_election.cc:304] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.705507 2052 raft_consensus.cc:2747] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.708899 1154 tablet_bootstrap.cc:492] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.709007 1154 ts_tablet_manager.cc:1397] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.066s user 0.022s sys 0.009s
I20250629 01:51:54.709575 1154 raft_consensus.cc:357] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.709708 1154 raft_consensus.cc:383] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.709755 1154 raft_consensus.cc:738] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.709928 1154 consensus_queue.cc:260] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.710341 1154 ts_tablet_manager.cc:1428] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:54.710753 1154 tablet_bootstrap.cc:492] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.711970 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8715038ee85a4103b95aea0f5f85b2e6" } new_table_name: "default.Table_a768996f666e43669ed0128c8dde5429" modify_external_catalogs: true
I20250629 01:51:54.714021 1154 tablet_bootstrap.cc:654] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.715386 455 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:54.716943 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 899) took 1530 ms. Trace:
I20250629 01:51:54.717013 1157 rpcz_store.cc:276] 0629 01:51:53.186660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.186831 (+ 171us) service_pool.cc:225] Handling call
0629 01:51:54.716840 (+1530009us) ts_tablet_manager.cc:1145] Deleting tablet 996ad157d7804856bdb9b606bd412710
0629 01:51:54.716848 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.716928 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529876}
I20250629 01:51:54.717339 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 900) took 1521 ms. Trace:
I20250629 01:51:54.717403 1157 rpcz_store.cc:276] 0629 01:51:53.196179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.202122 (+ 5943us) service_pool.cc:225] Handling call
0629 01:51:54.717247 (+1515125us) ts_tablet_manager.cc:1145] Deleting tablet 54f38ccc485043ba8b8a6f644a3e4f36
0629 01:51:54.717255 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.717326 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1514920}
I20250629 01:51:54.717609 1157 tablet_replica.cc:331] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:54.717672 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 996ad157d7804856bdb9b606bd412710 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 996ad157d7804856bdb9b606bd412710 already in progress: creating tablet
I20250629 01:51:54.717739 1157 raft_consensus.cc:2241] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.717852 1157 raft_consensus.cc:2270] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.717900 1156 ts_tablet_manager.cc:1918] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:54.718068 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 54f38ccc485043ba8b8a6f644a3e4f36 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54f38ccc485043ba8b8a6f644a3e4f36 already in progress: creating tablet
I20250629 01:51:54.718173 1156 log.cc:1199] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f5941dd6381041538be6c92858d6ba13
I20250629 01:51:54.718258 2052 raft_consensus.cc:491] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.718295 1157 ts_tablet_manager.cc:1905] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.718420 1156 ts_tablet_manager.cc:1939] T f5941dd6381041538be6c92858d6ba13 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.718353 2052 raft_consensus.cc:513] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.718883 2052 leader_election.cc:290] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.719607 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f5941dd6381041538be6c92858d6ba13 (table default.Table_475fe7b17a5c4aac90faf6432e488d87 [id=0fb1390e9626454eba8e8d5c634e3943]) successfully deleted
I20250629 01:51:54.719717 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56b9ef027c9d42048e4a6dff221df793" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.720235 1156 tablet_replica.cc:331] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:54.720518 712 leader_election.cc:343] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.722656 455 catalog_manager.cc:5582] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:54.723019 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]). Expected version 1 got 0
I20250629 01:51:54.723227 1157 ts_tablet_manager.cc:1918] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.723242 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
I20250629 01:51:54.723304 1157 log.cc:1199] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/77eae224763d464496066b4a98efe5ff
I20250629 01:51:54.723526 1157 ts_tablet_manager.cc:1939] T 77eae224763d464496066b4a98efe5ff P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:54.724431 710 leader_election.cc:343] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 56b9ef027c9d42048e4a6dff221df793
I20250629 01:51:54.724511 710 leader_election.cc:304] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.724805 2052 raft_consensus.cc:2747] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.726568 1156 raft_consensus.cc:2241] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.726982 1156 raft_consensus.cc:2270] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.727533 1156 ts_tablet_manager.cc:1905] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.728041 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_51f8fff3136f4bdeb43af63ea621a31e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.728471 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51f8fff3136f4bdeb43af63ea621a31e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.735517 1156 ts_tablet_manager.cc:1918] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.737872 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0dff81ebac6447489df5d50eea2401ea" } new_table_name: "default.Table_adcbb5e4f5cb4ae78e1f48f042c4033e" modify_external_catalogs: true
I20250629 01:51:54.738359 1156 log.cc:1199] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f6d79ac3c5f04b5db295c93c3aae92cb
I20250629 01:51:54.738739 1156 ts_tablet_manager.cc:1939] T f6d79ac3c5f04b5db295c93c3aae92cb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.723743 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 902) took 1514 ms. Trace:
I20250629 01:51:54.739198 1157 rpcz_store.cc:276] 0629 01:51:53.209529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.209689 (+ 160us) service_pool.cc:225] Handling call
0629 01:51:54.717577 (+1507888us) ts_tablet_manager.cc:1145] Deleting tablet 77eae224763d464496066b4a98efe5ff
0629 01:51:54.717584 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.720796 (+ 3212us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.723166 (+ 2370us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.723730 (+ 564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507744}
I20250629 01:51:54.739643 1157 tablet_replica.cc:331] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.739976 1157 raft_consensus.cc:2241] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.740293 1157 raft_consensus.cc:2270] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.740732 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f6d79ac3c5f04b5db295c93c3aae92cb (table default.Table_7eb487cf3b984b829ea5456c141ea824 [id=16fca49978904e5c880caa5d56572c22]) successfully deleted
I20250629 01:51:54.740867 1157 ts_tablet_manager.cc:1905] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.741035 1156 tablet_replica.cc:331] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.741364 1156 raft_consensus.cc:2241] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.741693 1156 raft_consensus.cc:2270] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.742249 1156 ts_tablet_manager.cc:1905] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.744745 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 77eae224763d464496066b4a98efe5ff (table default.Table_6f7d1b03023842daa7f9d43483dc6b95 [id=92be86eee6b64fb0984d8e5ea727fbdf]) successfully deleted
I20250629 01:51:54.746774 1154 tablet_bootstrap.cc:492] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.746937 1154 ts_tablet_manager.cc:1397] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20250629 01:51:54.747637 1154 raft_consensus.cc:357] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.747823 1154 raft_consensus.cc:383] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.747923 1154 raft_consensus.cc:738] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.748148 1154 consensus_queue.cc:260] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.749615 1156 ts_tablet_manager.cc:1918] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.751997 888 tablet_service.cc:1468] Processing CreateTablet for tablet 28ded2ad118049e096d62a3ba71d9b80 (DEFAULT_TABLE table=default.Table_6864147ee415442093dab8c36960aa39 [id=3a4788f1ded642fb811a8c7bc9897b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.752348 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28ded2ad118049e096d62a3ba71d9b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.753185 1154 ts_tablet_manager.cc:1428] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:51:54.753362 881 tablet_service.cc:1468] Processing CreateTablet for tablet 43624b0c909647f49981fff8584c956f (DEFAULT_TABLE table=default.Table_fd06736c67aa446890bfcc17305ecf16 [id=b3797cc064bf4c84b8aee9a8db547844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.753621 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43624b0c909647f49981fff8584c956f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.756358 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 28ded2ad118049e096d62a3ba71d9b80 (DEFAULT_TABLE table=default.Table_6864147ee415442093dab8c36960aa39 [id=3a4788f1ded642fb811a8c7bc9897b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.756649 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28ded2ad118049e096d62a3ba71d9b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.757586 1154 tablet_bootstrap.cc:492] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.758950 1154 tablet_bootstrap.cc:654] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.760588 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 43624b0c909647f49981fff8584c956f (DEFAULT_TABLE table=default.Table_fd06736c67aa446890bfcc17305ecf16 [id=b3797cc064bf4c84b8aee9a8db547844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.760851 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43624b0c909647f49981fff8584c956f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.761221 1159 tablet_bootstrap.cc:492] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.761309 1159 ts_tablet_manager.cc:1397] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.116s user 0.000s sys 0.004s
I20250629 01:51:54.761884 1159 raft_consensus.cc:357] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.762001 1159 raft_consensus.cc:383] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.762040 1159 raft_consensus.cc:738] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.762185 1159 consensus_queue.cc:260] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.762540 1159 ts_tablet_manager.cc:1428] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:54.762748 1159 tablet_bootstrap.cc:492] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:54.763983 1159 tablet_bootstrap.cc:654] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.765376 1157 ts_tablet_manager.cc:1918] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.767011 1157 log.cc:1199] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/69f0ae4c51924370896c624affde6761
I20250629 01:51:54.767299 1157 ts_tablet_manager.cc:1939] T 69f0ae4c51924370896c624affde6761 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.767836 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 903) took 1558 ms. Trace:
I20250629 01:51:54.769088 1157 rpcz_store.cc:276] 0629 01:51:53.209644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.210349 (+ 705us) service_pool.cc:225] Handling call
0629 01:51:54.739605 (+1529256us) ts_tablet_manager.cc:1145] Deleting tablet 69f0ae4c51924370896c624affde6761
0629 01:51:54.739613 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.760923 (+ 21310us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.765316 (+ 4393us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.767825 (+ 2509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529103}
I20250629 01:51:54.769532 1157 tablet_replica.cc:331] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.769820 1157 raft_consensus.cc:2241] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:54.770241 1157 raft_consensus.cc:2270] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.770949 1157 ts_tablet_manager.cc:1905] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.771112 2217 raft_consensus.cc:491] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.773088 2217 raft_consensus.cc:513] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.773701 2217 leader_election.cc:290] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.778623 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 69f0ae4c51924370896c624affde6761 (table default.Table_becfa32b77bb40b1b9df61126ffe628e [id=1c654e447e1b46f1a4570b41df8b1861]) successfully deleted
W20250629 01:51:54.778986 850 leader_election.cc:343] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:54.780397 2217 raft_consensus.cc:491] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.780493 2217 raft_consensus.cc:513] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.780685 1154 tablet_bootstrap.cc:492] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.780778 1154 ts_tablet_manager.cc:1397] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250629 01:51:54.780913 2217 leader_election.cc:290] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:54.782024 1154 raft_consensus.cc:357] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.782153 1154 raft_consensus.cc:383] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.782198 1154 raft_consensus.cc:738] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.782377 1154 consensus_queue.cc:260] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.782835 1154 ts_tablet_manager.cc:1428] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250629 01:51:54.782797 850 leader_election.cc:343] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 35990d206ba64159ab05ab7b7b19d064
W20250629 01:51:54.783021 850 leader_election.cc:343] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 35990d206ba64159ab05ab7b7b19d064
I20250629 01:51:54.783174 850 leader_election.cc:304] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.783525 2217 raft_consensus.cc:2747] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.783721 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:54.784312 1154 tablet_bootstrap.cc:492] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.787405 1154 tablet_bootstrap.cc:654] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.789824 1157 ts_tablet_manager.cc:1918] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:54.790062 1157 log.cc:1199] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/7d7b8cf118eb4a5c997815b97b9b7d00
I20250629 01:51:54.790477 1157 ts_tablet_manager.cc:1939] T 7d7b8cf118eb4a5c997815b97b9b7d00 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:54.790500 850 leader_election.cc:343] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:54.792218 850 leader_election.cc:304] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.792562 2217 raft_consensus.cc:2747] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.792960 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 901) took 1583 ms. Trace:
I20250629 01:51:54.764315 743 tablet_service.cc:1468] Processing CreateTablet for tablet 28ded2ad118049e096d62a3ba71d9b80 (DEFAULT_TABLE table=default.Table_6864147ee415442093dab8c36960aa39 [id=3a4788f1ded642fb811a8c7bc9897b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.793355 1157 rpcz_store.cc:276] 0629 01:51:53.209253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.211547 (+ 2294us) service_pool.cc:225] Handling call
0629 01:51:54.769500 (+1557953us) ts_tablet_manager.cc:1145] Deleting tablet 7d7b8cf118eb4a5c997815b97b9b7d00
0629 01:51:54.769508 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.783326 (+ 13818us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.789762 (+ 6436us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.792947 (+ 3185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548933}
I20250629 01:51:54.793572 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28ded2ad118049e096d62a3ba71d9b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.793697 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 7d7b8cf118eb4a5c997815b97b9b7d00 (table default.Table_77b9f4dc97974a79af78d31e6c953f5c [id=e6d6685d64154ba8ab80acf601e83986]) successfully deleted
I20250629 01:51:54.794435 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 905) took 1558 ms. Trace:
I20250629 01:51:54.794493 1157 rpcz_store.cc:276] 0629 01:51:53.235667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.236032 (+ 365us) service_pool.cc:225] Handling call
0629 01:51:54.794340 (+1558308us) ts_tablet_manager.cc:1145] Deleting tablet e304f0bfe13846f29e6390d9904a4f57
0629 01:51:54.794348 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.794419 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1558154}
W20250629 01:51:54.795450 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e304f0bfe13846f29e6390d9904a4f57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e304f0bfe13846f29e6390d9904a4f57 already in progress: creating tablet
I20250629 01:51:54.795809 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 904) took 1560 ms. Trace:
I20250629 01:51:54.795868 1157 rpcz_store.cc:276] 0629 01:51:53.235033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.236545 (+ 1512us) service_pool.cc:225] Handling call
0629 01:51:54.795719 (+1559174us) ts_tablet_manager.cc:1145] Deleting tablet 90eef69522e54b12b849d94e56355921
0629 01:51:54.795726 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.795793 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532959}
W20250629 01:51:54.796315 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 90eef69522e54b12b849d94e56355921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90eef69522e54b12b849d94e56355921 already in progress: creating tablet
I20250629 01:51:54.772742 749 tablet_service.cc:1468] Processing CreateTablet for tablet 43624b0c909647f49981fff8584c956f (DEFAULT_TABLE table=default.Table_fd06736c67aa446890bfcc17305ecf16 [id=b3797cc064bf4c84b8aee9a8db547844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.799232 1157 tablet_replica.cc:331] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.799278 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43624b0c909647f49981fff8584c956f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.799690 1157 raft_consensus.cc:2241] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.799995 1157 raft_consensus.cc:2270] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.800566 1154 tablet_bootstrap.cc:492] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.800704 1154 ts_tablet_manager.cc:1397] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20250629 01:51:54.801324 1154 raft_consensus.cc:357] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.801486 1154 raft_consensus.cc:383] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.801568 1154 raft_consensus.cc:738] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.801810 1154 consensus_queue.cc:260] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.805680 1154 ts_tablet_manager.cc:1428] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:54.807053 1154 tablet_bootstrap.cc:492] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.808854 1154 tablet_bootstrap.cc:654] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.809867 1157 ts_tablet_manager.cc:1905] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.811585 2217 raft_consensus.cc:491] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.811693 2217 raft_consensus.cc:513] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.812189 2217 leader_election.cc:290] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.812486 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2b8e1cf9c4eb40c2ba24d1490398e140" } modify_external_catalogs: true
I20250629 01:51:54.812634 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2b8e1cf9c4eb40c2ba24d1490398e140" } modify_external_catalogs: true
W20250629 01:51:54.813896 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c90ec6864664fe48d8b9a92837b748c consensus state has no leader
I20250629 01:51:54.814273 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_29974721952f4bcba85323986edf64ac" } new_table_name: "default.Table_6c1810f26a0140c18dd11d5b769dcffd" modify_external_catalogs: true
I20250629 01:51:54.815421 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_660af79d2b29409fae344ba080c4519a" } new_table_name: "default.Table_396c6df1e4db40aeae8e6980af7acc13" modify_external_catalogs: true
W20250629 01:51:54.816381 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbe1f8083a884cdd97afce83bd22073c consensus state has no leader
I20250629 01:51:54.816632 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_11fa9f38439b4f6d990266584eebfaf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.816931 2052 raft_consensus.cc:491] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.817024 2052 raft_consensus.cc:513] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:54.817101 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11fa9f38439b4f6d990266584eebfaf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:54.818204 710 leader_election.cc:343] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:54.818320 2052 leader_election.cc:290] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.819401 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]). Expected version 2 got 0
W20250629 01:51:54.820276 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8bed6b19b8454766add0f458a1eac7c4 has no consensus state
I20250629 01:51:54.820510 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cf31c7a57f0b412894e2702dc66893fb" } modify_external_catalogs: true
I20250629 01:51:54.820619 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cf31c7a57f0b412894e2702dc66893fb" } modify_external_catalogs: true
I20250629 01:51:54.821007 1157 ts_tablet_manager.cc:1918] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.821082 1157 log.cc:1199] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b9b680859a1a4c56afe33f5b1b3c0793
I20250629 01:51:54.821312 1157 ts_tablet_manager.cc:1939] T b9b680859a1a4c56afe33f5b1b3c0793 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.821522 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 906) took 1551 ms. Trace:
I20250629 01:51:54.821581 1157 rpcz_store.cc:276] 0629 01:51:53.270154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.270197 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:54.799194 (+1528997us) ts_tablet_manager.cc:1145] Deleting tablet b9b680859a1a4c56afe33f5b1b3c0793
0629 01:51:54.799202 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.818587 (+ 19385us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.820946 (+ 2359us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.821510 (+ 564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528860}
I20250629 01:51:54.822310 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.822402 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 with delete_type TABLET_DATA_DELETED (replaced by tablet 35a62c3fe4fb4220838deece8e46d074) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.822476 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_768dd2c93bfa4a828037ed23b11dc5ae" } modify_external_catalogs: true
I20250629 01:51:54.822603 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_768dd2c93bfa4a828037ed23b11dc5ae" } modify_external_catalogs: true
W20250629 01:51:54.821995 850 leader_election.cc:343] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:54.823947 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3efca07702364cf982f01059b4d3899c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.824148 850 leader_election.cc:343] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:54.824224 850 leader_election.cc:304] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:54.824426 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3efca07702364cf982f01059b4d3899c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.824690 2217 raft_consensus.cc:2747] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.825124 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 907) took 1542 ms. Trace:
I20250629 01:51:54.825419 1157 rpcz_store.cc:276] 0629 01:51:53.282352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.282391 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:54.825038 (+1542647us) ts_tablet_manager.cc:1145] Deleting tablet 0f05d54fc9e649faabee85a418ebf2bb
0629 01:51:54.825046 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.825111 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542505}
I20250629 01:51:54.825997 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 908) took 1542 ms. Trace:
I20250629 01:51:54.826303 1157 rpcz_store.cc:276] 0629 01:51:53.283608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.283650 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:54.825912 (+1542262us) ts_tablet_manager.cc:1145] Deleting tablet 1e4a173d9e70453c91d8ff79a710bc00
0629 01:51:54.825920 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.825985 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542114}
I20250629 01:51:54.827283 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b9b680859a1a4c56afe33f5b1b3c0793 (table default.Table_30467a20b5b54ca7894c405b22705bc2 [id=66af0a365c3f40deac642473f9125b10]) successfully deleted
W20250629 01:51:54.827498 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0f05d54fc9e649faabee85a418ebf2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f05d54fc9e649faabee85a418ebf2bb already in progress: creating tablet
W20250629 01:51:54.827819 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1e4a173d9e70453c91d8ff79a710bc00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e4a173d9e70453c91d8ff79a710bc00 already in progress: creating tablet
I20250629 01:51:54.827833 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 909) took 1499 ms. Trace:
I20250629 01:51:54.828136 1157 rpcz_store.cc:276] 0629 01:51:53.328496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.328542 (+ 46us) service_pool.cc:225] Handling call
0629 01:51:54.827751 (+1499209us) ts_tablet_manager.cc:1145] Deleting tablet 19b642dd58e948329442964df736b4a4
0629 01:51:54.827758 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.827820 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499066}
W20250629 01:51:54.828747 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 19b642dd58e948329442964df736b4a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b642dd58e948329442964df736b4a4 already in progress: creating tablet
I20250629 01:51:54.829053 1157 tablet_replica.cc:331] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.829237 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 942f7bbe29f94083b2494aab8fbcc0c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.829466 1157 raft_consensus.cc:2241] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.829744 1157 raft_consensus.cc:2270] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.830750 1157 ts_tablet_manager.cc:1905] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.831346 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.831657 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 35990d206ba64159ab05ab7b7b19d064 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.832763 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5d2f2975e5d64efd888ed948f1420916
I20250629 01:51:54.833161 451 catalog_manager.cc:6652] ReplaceTablet: tablet 5d2f2975e5d64efd888ed948f1420916 deleted and replaced by tablet a1a88bf7668546f4b303dcddba10d1a0
W20250629 01:51:54.834156 712 consensus_peers.cc:489] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:54.834156 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d2f2975e5d64efd888ed948f1420916 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a88bf7668546f4b303dcddba10d1a0) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.834306 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d2f2975e5d64efd888ed948f1420916 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a88bf7668546f4b303dcddba10d1a0) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.834316 1050 raft_consensus.cc:1273] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:54.834380 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5d2f2975e5d64efd888ed948f1420916 with delete_type TABLET_DATA_DELETED (replaced by tablet a1a88bf7668546f4b303dcddba10d1a0) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.820026 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71d82a98d0124d08a3ac0fce4fb47062" } modify_external_catalogs: true
I20250629 01:51:54.834501 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71d82a98d0124d08a3ac0fce4fb47062" } modify_external_catalogs: true
I20250629 01:51:54.834896 2052 consensus_queue.cc:1035] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:54.835651 1158 tablet_replica.cc:331] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.835798 1158 raft_consensus.cc:2241] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.835922 1158 raft_consensus.cc:2270] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.836284 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8e2c9df275d2453daa8b73a723d56ed6" } modify_external_catalogs: true
I20250629 01:51:54.836428 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8e2c9df275d2453daa8b73a723d56ed6" } modify_external_catalogs: true
I20250629 01:51:54.836705 1158 ts_tablet_manager.cc:1905] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.841724 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.842993 1157 ts_tablet_manager.cc:1918] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.843809 1158 ts_tablet_manager.cc:1918] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.843963 1158 log.cc:1199] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/648b0a0f742b4e7da9eaff6bd8795fc4
I20250629 01:51:54.844450 1158 ts_tablet_manager.cc:1939] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.845547 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 648b0a0f742b4e7da9eaff6bd8795fc4 (table default.Table_afb98d5207ae4d2d8b97b8577ae3a157 [id=e874833b32de4076921fdf0e2f103a1a]) successfully deleted
W20250629 01:51:54.846375 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5d2f2975e5d64efd888ed948f1420916 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2f2975e5d64efd888ed948f1420916 already in progress: creating tablet
I20250629 01:51:54.846961 1158 tablet_replica.cc:331] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:54.847322 1158 raft_consensus.cc:2241] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.847633 1158 raft_consensus.cc:2270] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.849290 1156 log.cc:1199] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ef976703d61442f8987ff32070a33592
I20250629 01:51:54.849613 1156 ts_tablet_manager.cc:1939] T ef976703d61442f8987ff32070a33592 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:54.850692 1157 log.cc:1199] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0827ec622ff04ca6b2fec50e40b9e1de
I20250629 01:51:54.848147 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3db333c60cb3485a9daae316b9f1a8c8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.850955 1158 ts_tablet_manager.cc:1905] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.850338 2217 raft_consensus.cc:491] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.851475 1157 ts_tablet_manager.cc:1939] T 0827ec622ff04ca6b2fec50e40b9e1de P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.851022 888 tablet_service.cc:1468] Processing CreateTablet for tablet 62d0c31621124a5baaf0a1cb436f5f44 (DEFAULT_TABLE table=default.Table_29974721952f4bcba85323986edf64ac [id=aeb4cc0466a9499ab2321d475fe8ed83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.851408 2217 raft_consensus.cc:513] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.851730 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d0c31621124a5baaf0a1cb436f5f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.847112 1159 tablet_bootstrap.cc:492] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.851842 1159 ts_tablet_manager.cc:1397] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.009s
I20250629 01:51:54.851923 2217 leader_election.cc:290] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.852581 1159 raft_consensus.cc:357] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.852756 1159 raft_consensus.cc:383] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.852835 1159 raft_consensus.cc:738] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.853067 1159 consensus_queue.cc:260] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
W20250629 01:51:54.853492 850 leader_election.cc:343] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:54.853649 1159 ts_tablet_manager.cc:1428] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:54.854081 1159 tablet_bootstrap.cc:492] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:54.854496 850 leader_election.cc:343] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:54.854586 850 leader_election.cc:304] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:54.854875 2217 raft_consensus.cc:2747] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.851711 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 912) took 1437 ms. Trace:
I20250629 01:51:54.856318 1157 rpcz_store.cc:276] 0629 01:51:53.414390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.414683 (+ 293us) service_pool.cc:225] Handling call
0629 01:51:54.829021 (+1414338us) ts_tablet_manager.cc:1145] Deleting tablet 0827ec622ff04ca6b2fec50e40b9e1de
0629 01:51:54.829028 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.833819 (+ 4791us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.842932 (+ 9113us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.851699 (+ 8767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414189}
I20250629 01:51:54.857439 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b0da5a36b5c141cc848f23ee43a30fa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:54.857601 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0827ec622ff04ca6b2fec50e40b9e1de (table default.Table_8c1fbbaabe004466858cce35b18194b1 [id=700ef08e094b44369154076e65d9ea94]) successfully deleted
I20250629 01:51:54.857860 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 914) took 1396 ms. Trace:
I20250629 01:51:54.857939 1157 rpcz_store.cc:276] 0629 01:51:53.461476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.461529 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:54.857771 (+1396242us) ts_tablet_manager.cc:1145] Deleting tablet a0769f77ea234dd7bccc2244c08b512b
0629 01:51:54.857780 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.857846 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1396093}
W20250629 01:51:54.858276 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a0769f77ea234dd7bccc2244c08b512b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0769f77ea234dd7bccc2244c08b512b already in progress: creating tablet
W20250629 01:51:54.858327 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0da5a36b5c141cc848f23ee43a30fa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.859032 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 913) took 1398 ms. Trace:
I20250629 01:51:54.859104 1157 rpcz_store.cc:276] 0629 01:51:53.460880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.463112 (+ 2232us) service_pool.cc:225] Handling call
0629 01:51:54.858949 (+1395837us) ts_tablet_manager.cc:1145] Deleting tablet a0002dca4db24405afdce3e1076eb52e
0629 01:51:54.858957 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.859019 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1395679}
W20250629 01:51:54.859426 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a0002dca4db24405afdce3e1076eb52e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0002dca4db24405afdce3e1076eb52e already in progress: creating tablet
I20250629 01:51:54.859966 1159 tablet_bootstrap.cc:654] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.860715 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 62d0c31621124a5baaf0a1cb436f5f44 (DEFAULT_TABLE table=default.Table_29974721952f4bcba85323986edf64ac [id=aeb4cc0466a9499ab2321d475fe8ed83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.861011 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d0c31621124a5baaf0a1cb436f5f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.861163 743 tablet_service.cc:1468] Processing CreateTablet for tablet 62d0c31621124a5baaf0a1cb436f5f44 (DEFAULT_TABLE table=default.Table_29974721952f4bcba85323986edf64ac [id=aeb4cc0466a9499ab2321d475fe8ed83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.861457 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d0c31621124a5baaf0a1cb436f5f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.860980 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 916) took 1364 ms. Trace:
I20250629 01:51:54.861874 1157 rpcz_store.cc:276] 0629 01:51:53.496316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.496373 (+ 57us) service_pool.cc:225] Handling call
0629 01:51:54.860890 (+1364517us) ts_tablet_manager.cc:1145] Deleting tablet 07d5eb1b3c3345c099f298df240a5c75
0629 01:51:54.860905 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.860966 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364335}
W20250629 01:51:54.862213 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 07d5eb1b3c3345c099f298df240a5c75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07d5eb1b3c3345c099f298df240a5c75 already in progress: creating tablet
I20250629 01:51:54.862637 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 915) took 1366 ms. Trace:
I20250629 01:51:54.862699 1157 rpcz_store.cc:276] 0629 01:51:53.496167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.501367 (+ 5200us) service_pool.cc:225] Handling call
0629 01:51:54.862560 (+1361193us) ts_tablet_manager.cc:1145] Deleting tablet 06e31520d3ad4c3395a59049755a4725
0629 01:51:54.862567 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.862624 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360952}
W20250629 01:51:54.863018 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 06e31520d3ad4c3395a59049755a4725 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06e31520d3ad4c3395a59049755a4725 already in progress: creating tablet
I20250629 01:51:54.864111 1157 tablet_replica.cc:331] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.864249 1157 raft_consensus.cc:2241] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.864362 1157 raft_consensus.cc:2270] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.864748 1157 ts_tablet_manager.cc:1905] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.867848 888 tablet_service.cc:1468] Processing CreateTablet for tablet db110b800da44aafa035b4286fe9a985 (DEFAULT_TABLE table=default.Table_5f0f9ca3a65845f18135c7c9a9dd3991 [id=188d48732bd543398dc0364a4e8a0dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.868196 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db110b800da44aafa035b4286fe9a985. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:54.870757 712 leader_election.cc:343] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.870849 712 leader_election.cc:304] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.871202 2228 raft_consensus.cc:2747] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.875526 2238 tablet.cc:1722] T ddc24d0e668c49ac825b2578849ab2a3 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:54.875839 2231 mvcc.cc:204] Tried to move back new op lower bound from 7172759203152596992 to 7172759201239957504. Current Snapshot: MvccSnapshot[applied={T|T < 7172759203152596992}]
W20250629 01:51:54.876158 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ee1e7985f4b4422ba607962bf1fdc42 consensus state has no leader
I20250629 01:51:54.877759 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0501e4a60a1046ad958d2d1d4e077bc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.878152 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0501e4a60a1046ad958d2d1d4e077bc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:54.879027 452 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet ec35262494e54e9380722971cd4dc43c
I20250629 01:51:54.879289 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3c3ba33385624970b8f09248b76e5cc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:54.879671 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c3ba33385624970b8f09248b76e5cc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:54.880836 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62d0c31621124a5baaf0a1cb436f5f44 consensus state has no leader
I20250629 01:51:54.881732 881 tablet_service.cc:1515] Processing DeleteTablet for tablet e962a020664b42ea918e5952dc43aade with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.881851 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 60e12d28ef784d898dbf9846f7063fdf
I20250629 01:51:54.882929 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 60e12d28ef784d898dbf9846f7063fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.883535 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e332a512574e4b4e881aeb37e289f0bc
I20250629 01:51:54.884532 881 tablet_service.cc:1515] Processing DeleteTablet for tablet e332a512574e4b4e881aeb37e289f0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.884552 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e332a512574e4b4e881aeb37e289f0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.884765 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e332a512574e4b4e881aeb37e289f0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.886651 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet ddc24d0e668c49ac825b2578849ab2a3 (table default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e]). Expected version 2 got 0
I20250629 01:51:54.887395 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a53d9e8b7efe46099821ae628640a42e
I20250629 01:51:54.888537 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a53d9e8b7efe46099821ae628640a42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.889000 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a53d9e8b7efe46099821ae628640a42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.889149 2049 raft_consensus.cc:491] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.889254 2049 raft_consensus.cc:513] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.889705 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 60e12d28ef784d898dbf9846f7063fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.889744 2049 leader_election.cc:290] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.889163 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a53d9e8b7efe46099821ae628640a42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.890043 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 60e12d28ef784d898dbf9846f7063fdf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.891258 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e5d27d658892496a9b19b57949a8399a
W20250629 01:51:54.891446 710 consensus_peers.cc:489] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 798516e55ca747fcb2942026adb78af0. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:54.891939 881 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:54.892530 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:54.895680 1157 ts_tablet_manager.cc:1918] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.897782 1988 mvcc.cc:204] Tried to move back new op lower bound from 7172759203152596992 to 7172759201239957504. Current Snapshot: MvccSnapshot[applied={T|T < 7172759203152596992}]
I20250629 01:51:54.897925 2233 tablet.cc:1722] T ddc24d0e668c49ac825b2578849ab2a3 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:54.898344 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ef976703d61442f8987ff32070a33592 (table default.Table_f220872b43b3433694c2f522f6e54acd [id=b1662df1e42f43eeae59f02a3e446006]) successfully deleted
I20250629 01:51:54.898849 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:54.899672 1156 tablet_replica.cc:331] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:54.899837 1156 raft_consensus.cc:2241] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.900259 1026 tablet_service.cc:1468] Processing CreateTablet for tablet db110b800da44aafa035b4286fe9a985 (DEFAULT_TABLE table=default.Table_5f0f9ca3a65845f18135c7c9a9dd3991 [id=188d48732bd543398dc0364a4e8a0dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.900765 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db110b800da44aafa035b4286fe9a985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.902472 743 tablet_service.cc:1468] Processing CreateTablet for tablet db110b800da44aafa035b4286fe9a985 (DEFAULT_TABLE table=default.Table_5f0f9ca3a65845f18135c7c9a9dd3991 [id=188d48732bd543398dc0364a4e8a0dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.902810 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db110b800da44aafa035b4286fe9a985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.903215 1156 raft_consensus.cc:2270] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.903393 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6bf3c7d0bb548e0b77ba8b569a72d0d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.903650 1156 ts_tablet_manager.cc:1905] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.903803 1050 raft_consensus.cc:2466] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.904790 2228 raft_consensus.cc:491] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.904939 2228 raft_consensus.cc:513] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.906616 2228 leader_election.cc:290] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.906208 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 8873b53d7e42492da64f7f677e05c6f1 (DEFAULT_TABLE table=default.Table_b2805bb726134583891dea5763bcb7c1 [id=d3f00a0302b24c4ba358c631e65ae965]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.906754 1158 ts_tablet_manager.cc:1918] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.906970 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8873b53d7e42492da64f7f677e05c6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.907243 1158 log.cc:1199] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/2845d8fbf3f34c769027b9b6bb3e1f08
I20250629 01:51:54.908962 1158 ts_tablet_manager.cc:1939] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:54.912386 750 tablet_service.cc:1468] Processing CreateTablet for tablet 8873b53d7e42492da64f7f677e05c6f1 (DEFAULT_TABLE table=default.Table_b2805bb726134583891dea5763bcb7c1 [id=d3f00a0302b24c4ba358c631e65ae965]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.912676 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8873b53d7e42492da64f7f677e05c6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.906433 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:54.918213 1048 raft_consensus.cc:2466] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:54.919004 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 2845d8fbf3f34c769027b9b6bb3e1f08 (table default.Table_208f2046cc46452887e343bab242e4b2 [id=77c157c54c864ddbb4678e9af6b1fe43]) successfully deleted
I20250629 01:51:54.919032 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c7deaf4d5b1b442cb63fde7b7e42ad6b (DEFAULT_TABLE table=default.Table_b02ba93133744171884cf63996d45eb0 [id=7eb7e11ce0624f1293050ff13d1f621e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:54.919195 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e332a512574e4b4e881aeb37e289f0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e332a512574e4b4e881aeb37e289f0bc already in progress: creating tablet
I20250629 01:51:54.919183 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 28da32ab17024f519105c2490efd9136 (DEFAULT_TABLE table=default.Table_e068250f05fd4e77bf37790a635a1eb6 [id=98ca76427dba4c60acd108dc144ca668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.919312 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7deaf4d5b1b442cb63fde7b7e42ad6b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:54.919389 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a53d9e8b7efe46099821ae628640a42e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53d9e8b7efe46099821ae628640a42e already in progress: creating tablet
I20250629 01:51:54.919492 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28da32ab17024f519105c2490efd9136. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:54.919550 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 60e12d28ef784d898dbf9846f7063fdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60e12d28ef784d898dbf9846f7063fdf already in progress: creating tablet
W20250629 01:51:54.919739 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e5d27d658892496a9b19b57949a8399a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5d27d658892496a9b19b57949a8399a already in progress: creating tablet
I20250629 01:51:54.919718 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 27b76e04158e492a9979c06e7cc2ae37 (DEFAULT_TABLE table=default.Table_e5ccc8748dbe46e697a27a8368252d15 [id=686d0a50e34941239140d0d7c83a2ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.920038 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b76e04158e492a9979c06e7cc2ae37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.920639 1157 log.cc:1199] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/ea00b4559e03410885180286b5e9fd10
I20250629 01:51:54.920622 755 tablet_service.cc:1468] Processing CreateTablet for tablet c7deaf4d5b1b442cb63fde7b7e42ad6b (DEFAULT_TABLE table=default.Table_b02ba93133744171884cf63996d45eb0 [id=7eb7e11ce0624f1293050ff13d1f621e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.920881 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7deaf4d5b1b442cb63fde7b7e42ad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.920893 1157 ts_tablet_manager.cc:1939] T ea00b4559e03410885180286b5e9fd10 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.921118 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 917) took 1392 ms. Trace:
I20250629 01:51:54.921176 1157 rpcz_store.cc:276] 0629 01:51:53.528247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.528568 (+ 321us) service_pool.cc:225] Handling call
0629 01:51:54.864061 (+1335493us) ts_tablet_manager.cc:1145] Deleting tablet ea00b4559e03410885180286b5e9fd10
0629 01:51:54.864069 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.893214 (+ 29145us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.895618 (+ 2404us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.921107 (+ 25489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1335344}
I20250629 01:51:54.923393 745 tablet_service.cc:1468] Processing CreateTablet for tablet 27b76e04158e492a9979c06e7cc2ae37 (DEFAULT_TABLE table=default.Table_e5ccc8748dbe46e697a27a8368252d15 [id=686d0a50e34941239140d0d7c83a2ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.923645 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b76e04158e492a9979c06e7cc2ae37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.924294 738 tablet_service.cc:1468] Processing CreateTablet for tablet 28da32ab17024f519105c2490efd9136 (DEFAULT_TABLE table=default.Table_e068250f05fd4e77bf37790a635a1eb6 [id=98ca76427dba4c60acd108dc144ca668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.924561 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28da32ab17024f519105c2490efd9136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.924835 710 leader_election.cc:304] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.904641 881 tablet_service.cc:1468] Processing CreateTablet for tablet 8873b53d7e42492da64f7f677e05c6f1 (DEFAULT_TABLE table=default.Table_b2805bb726134583891dea5763bcb7c1 [id=d3f00a0302b24c4ba358c631e65ae965]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.905754 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.912070 896 tablet_service.cc:1468] Processing CreateTablet for tablet c7deaf4d5b1b442cb63fde7b7e42ad6b (DEFAULT_TABLE table=default.Table_b02ba93133744171884cf63996d45eb0 [id=7eb7e11ce0624f1293050ff13d1f621e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.928324 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8873b53d7e42492da64f7f677e05c6f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.928296 740 tablet_service.cc:1468] Processing CreateTablet for tablet cace39b5f01f4cf79b21c18b72091dbd (DEFAULT_TABLE table=default.Table_a1dcfdd16d0746d0a7ff69766d0b967d [id=190daae95baa41a380fa13184b628885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.928491 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7deaf4d5b1b442cb63fde7b7e42ad6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.928532 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cace39b5f01f4cf79b21c18b72091dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.913139 888 tablet_service.cc:1468] Processing CreateTablet for tablet 28da32ab17024f519105c2490efd9136 (DEFAULT_TABLE table=default.Table_e068250f05fd4e77bf37790a635a1eb6 [id=98ca76427dba4c60acd108dc144ca668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.935015 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 918) took 1398 ms. Trace:
I20250629 01:51:54.935091 1157 rpcz_store.cc:276] 0629 01:51:53.536436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.536480 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:54.934908 (+1398428us) ts_tablet_manager.cc:1145] Deleting tablet 250ce9c0b27c40d996db138d0acb28d7
0629 01:51:54.934916 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.935002 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1398003}
I20250629 01:51:54.915083 1159 tablet_bootstrap.cc:492] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:54.917408 895 tablet_service.cc:1468] Processing CreateTablet for tablet 27b76e04158e492a9979c06e7cc2ae37 (DEFAULT_TABLE table=default.Table_e5ccc8748dbe46e697a27a8368252d15 [id=686d0a50e34941239140d0d7c83a2ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.935514 1159 ts_tablet_manager.cc:1397] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.081s user 0.002s sys 0.005s
I20250629 01:51:54.918488 890 tablet_service.cc:1468] Processing CreateTablet for tablet cace39b5f01f4cf79b21c18b72091dbd (DEFAULT_TABLE table=default.Table_a1dcfdd16d0746d0a7ff69766d0b967d [id=190daae95baa41a380fa13184b628885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.935931 2052 raft_consensus.cc:2802] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.935999 2052 raft_consensus.cc:491] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.936051 2052 raft_consensus.cc:3058] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.936157 1159 raft_consensus.cc:357] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:54.936245 1032 tablet_service.cc:1468] Processing CreateTablet for tablet cace39b5f01f4cf79b21c18b72091dbd (DEFAULT_TABLE table=default.Table_a1dcfdd16d0746d0a7ff69766d0b967d [id=190daae95baa41a380fa13184b628885]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.936529 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cace39b5f01f4cf79b21c18b72091dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.938066 2052 raft_consensus.cc:513] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.938498 2052 leader_election.cc:290] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.939441 1157 tablet_replica.cc:331] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:54.939610 1157 raft_consensus.cc:2241] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:54.940704 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet ea00b4559e03410885180286b5e9fd10 (table default.Table_5d35d0525f924ed692497dbaf2287e8e [id=e7ad9b02d0f14bdfa3914844146a1e0b]) successfully deleted
I20250629 01:51:54.941107 1049 raft_consensus.cc:1273] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:54.941105 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 250ce9c0b27c40d996db138d0acb28d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 250ce9c0b27c40d996db138d0acb28d7 already in progress: creating tablet
I20250629 01:51:54.935317 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28da32ab17024f519105c2490efd9136. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.941697 2052 consensus_queue.cc:1035] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:54.941808 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cace39b5f01f4cf79b21c18b72091dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.943470 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b76e04158e492a9979c06e7cc2ae37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.944681 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.944869 1048 raft_consensus.cc:3058] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.939740 1157 raft_consensus.cc:2270] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:54.945999 1157 ts_tablet_manager.cc:1905] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:54.947387 1048 raft_consensus.cc:2466] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:54.948040 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:54.952129 712 leader_election.cc:343] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:54.952382 712 consensus_peers.cc:489] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:51:54.952489 712 leader_election.cc:343] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.956025 710 leader_election.cc:304] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:54.956338 2049 raft_consensus.cc:2802] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.956440 2049 raft_consensus.cc:695] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.956681 2049 consensus_queue.cc:237] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.959404 1861 raft_consensus.cc:491] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:54.959429 710 leader_election.cc:304] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:54.959708 2052 raft_consensus.cc:2802] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:54.959769 2052 raft_consensus.cc:491] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:54.959820 2052 raft_consensus.cc:3058] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.959517 1861 raft_consensus.cc:513] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.960263 1861 leader_election.cc:290] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:54.960992 1155 tablet_bootstrap.cc:492] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:54.961133 1155 ts_tablet_manager.cc:1397] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.328s user 0.004s sys 0.029s
I20250629 01:51:54.961774 1155 raft_consensus.cc:357] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.961971 1155 raft_consensus.cc:383] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.962049 1155 raft_consensus.cc:738] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.962098 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f3a65c9c1c740b1865b96a6e2aea524" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:54.962301 1155 consensus_queue.cc:260] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:54.962443 712 leader_election.cc:343] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.962790 1155 ts_tablet_manager.cc:1428] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:54.963050 1155 tablet_bootstrap.cc:492] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:54.961779 2052 raft_consensus.cc:513] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.964471 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6bf3c7d0bb548e0b77ba8b569a72d0d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:54.964807 712 leader_election.cc:343] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:54.964938 2052 leader_election.cc:290] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:54.965845 1155 tablet_bootstrap.cc:654] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:54.968271 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6bf3c7d0bb548e0b77ba8b569a72d0d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:54.968431 1048 raft_consensus.cc:3058] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:54.968866 1157 ts_tablet_manager.cc:1918] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:54.968972 1157 log.cc:1199] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/2f7c1f1af39d47c191d34dc026f7f687
I20250629 01:51:54.969235 1157 ts_tablet_manager.cc:1939] T 2f7c1f1af39d47c191d34dc026f7f687 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:54.969477 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 920) took 1379 ms. Trace:
I20250629 01:51:54.969543 1157 rpcz_store.cc:276] 0629 01:51:53.590360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.590397 (+ 37us) service_pool.cc:225] Handling call
0629 01:51:54.939407 (+1349010us) ts_tablet_manager.cc:1145] Deleting tablet 2f7c1f1af39d47c191d34dc026f7f687
0629 01:51:54.939415 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.954999 (+ 15584us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.968804 (+ 13805us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:54.969464 (+ 660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348751}
I20250629 01:51:54.969920 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 2f7c1f1af39d47c191d34dc026f7f687 (table default.Table_f641040c5d3f4fabb4642ae6c32b10c7 [id=7607f62687244e718732520ce1342b1a]) successfully deleted
I20250629 01:51:54.970700 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:54.971745 1048 raft_consensus.cc:2466] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:51:54.971859 710 leader_election.cc:343] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 6f3a65c9c1c740b1865b96a6e2aea524
I20250629 01:51:54.971936 710 leader_election.cc:304] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:54.972384 710 leader_election.cc:304] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:54.972719 2052 raft_consensus.cc:2747] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:54.972885 2052 raft_consensus.cc:2802] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:54.972993 453 master_service.cc:948] ReplaceTablet: received request to replace tablet ac560295fddd44368e6f9bc543f7cbfa from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.973120 2052 raft_consensus.cc:695] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:54.973320 2052 consensus_queue.cc:237] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.975804 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 925) took 1304 ms. Trace:
I20250629 01:51:54.976150 1157 rpcz_store.cc:276] 0629 01:51:53.671538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.671669 (+ 131us) service_pool.cc:225] Handling call
0629 01:51:54.975700 (+1304031us) ts_tablet_manager.cc:1145] Deleting tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c
0629 01:51:54.975708 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.975786 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289500}
W20250629 01:51:54.976701 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60ede5ecb0f94ddfa3bfd3e7fbdbf39c already in progress: creating tablet
I20250629 01:51:54.978132 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 926) took 1300 ms. Trace:
I20250629 01:51:54.978461 1157 rpcz_store.cc:276] 0629 01:51:53.677289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.677339 (+ 50us) service_pool.cc:225] Handling call
0629 01:51:54.978026 (+1300687us) ts_tablet_manager.cc:1145] Deleting tablet 9cfff25bea764e62954e49611b71703c
0629 01:51:54.978034 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.978117 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1291777}
W20250629 01:51:54.979032 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9cfff25bea764e62954e49611b71703c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cfff25bea764e62954e49611b71703c already in progress: creating tablet
I20250629 01:51:54.979651 1154 tablet_bootstrap.cc:492] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.979794 1154 ts_tablet_manager.cc:1397] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.173s user 0.005s sys 0.017s
I20250629 01:51:54.980439 1154 raft_consensus.cc:357] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.980598 1154 raft_consensus.cc:383] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.980666 1154 raft_consensus.cc:738] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.980970 1154 consensus_queue.cc:260] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:54.981694 1154 ts_tablet_manager.cc:1428] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:54.981988 1154 tablet_bootstrap.cc:492] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.983409 1154 tablet_bootstrap.cc:654] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:54.984208 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 45461dc364d847d0b0272ac714559008 (table default.Table_72800b006f6e4b2eab6c57ecea71aef0 [id=53d17bca03a74467a681cb07cb85ef23], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet 45461dc364d847d0b0272ac714559008 consensus state has no leader
I20250629 01:51:54.987556 1050 raft_consensus.cc:1273] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:54.987847 712 consensus_peers.cc:489] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:54.987972 1861 consensus_queue.cc:1035] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:54.988590 1154 tablet_bootstrap.cc:492] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:54.989033 1154 ts_tablet_manager.cc:1397] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250629 01:51:54.989168 446 master_service.cc:948] ReplaceTablet: received request to replace tablet faca0e904a04456d9aead23abd44e00a from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:54.989740 1154 raft_consensus.cc:357] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.989908 1154 raft_consensus.cc:383] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:54.989991 1154 raft_consensus.cc:738] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:54.990195 1154 consensus_queue.cc:260] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:54.990576 745 tablet_service.cc:1468] Processing CreateTablet for tablet f6f82ee996c94196b912aace417715c9 (DEFAULT_TABLE table=default.Table_d17573a245c247849aa573b35b93e377 [id=cb20a9a97250475bb854c469ee22f8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.990684 1154 ts_tablet_manager.cc:1428] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:51:54.990904 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f82ee996c94196b912aace417715c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.990948 1154 tablet_bootstrap.cc:492] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:54.991238 890 tablet_service.cc:1468] Processing CreateTablet for tablet 0eaa38e09c874844aa2320a255684a3a (DEFAULT_TABLE table=default.Table_2c05f26e7f594408b830d3c545b58966 [id=2ed86e3d11d7479385ab44c255144265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.991554 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eaa38e09c874844aa2320a255684a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.992352 895 tablet_service.cc:1468] Processing CreateTablet for tablet 60857068ca3242a28a2ae1b8334102b9 (DEFAULT_TABLE table=default.Table_379aba83abb74b5f9264601fa00007d0 [id=d6229d509e87471a908e8e877dcb78e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.992671 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60857068ca3242a28a2ae1b8334102b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:54.995910 740 tablet_service.cc:1468] Processing CreateTablet for tablet 60857068ca3242a28a2ae1b8334102b9 (DEFAULT_TABLE table=default.Table_379aba83abb74b5f9264601fa00007d0 [id=d6229d509e87471a908e8e877dcb78e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:54.995976 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 935) took 1110 ms. Trace:
I20250629 01:51:54.996037 1157 rpcz_store.cc:276] 0629 01:51:53.885366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.885410 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:54.995864 (+1110454us) ts_tablet_manager.cc:1145] Deleting tablet daf312e60c014fe698d3ec9f1a1b769c
0629 01:51:54.995871 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.995961 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1110314}
I20250629 01:51:54.996215 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60857068ca3242a28a2ae1b8334102b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:54.996517 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet daf312e60c014fe698d3ec9f1a1b769c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daf312e60c014fe698d3ec9f1a1b769c already in progress: creating tablet
I20250629 01:51:54.999217 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 936) took 1083 ms. Trace:
I20250629 01:51:54.999404 1154 tablet_bootstrap.cc:654] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.000029 1157 rpcz_store.cc:276] 0629 01:51:53.915414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.915475 (+ 61us) service_pool.cc:225] Handling call
0629 01:51:54.999122 (+1083647us) ts_tablet_manager.cc:1145] Deleting tablet 74b3badd684d45fa88dc8cbcfc61001a
0629 01:51:54.999129 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.999204 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083489}
W20250629 01:51:55.000437 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 74b3badd684d45fa88dc8cbcfc61001a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74b3badd684d45fa88dc8cbcfc61001a already in progress: creating tablet
I20250629 01:51:55.001353 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 60857068ca3242a28a2ae1b8334102b9 (DEFAULT_TABLE table=default.Table_379aba83abb74b5f9264601fa00007d0 [id=d6229d509e87471a908e8e877dcb78e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.001657 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60857068ca3242a28a2ae1b8334102b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.001837 1157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 937) took 1075 ms. Trace:
I20250629 01:51:55.001967 1157 rpcz_store.cc:276] 0629 01:51:53.926195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:53.926255 (+ 60us) service_pool.cc:225] Handling call
0629 01:51:55.001742 (+1075487us) ts_tablet_manager.cc:1145] Deleting tablet d1131e0d9c06435fb732a8a72e421298
0629 01:51:55.001751 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.001823 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075315}
W20250629 01:51:55.002274 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d1131e0d9c06435fb732a8a72e421298 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1131e0d9c06435fb732a8a72e421298 already in progress: creating tablet
I20250629 01:51:55.002562 1157 tablet_replica.cc:331] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.002732 1157 raft_consensus.cc:2241] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.002832 1157 raft_consensus.cc:2270] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.003178 1861 raft_consensus.cc:491] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.003268 1861 raft_consensus.cc:513] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.003667 1861 leader_election.cc:290] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.004123 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b17172ab8b4e0dad36ebaac776dc03" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.004554 1157 ts_tablet_manager.cc:1905] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:55.004599 712 leader_election.cc:343] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.008322 1030 tablet_service.cc:1468] Processing CreateTablet for tablet f6f82ee996c94196b912aace417715c9 (DEFAULT_TABLE table=default.Table_d17573a245c247849aa573b35b93e377 [id=cb20a9a97250475bb854c469ee22f8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.008594 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f82ee996c94196b912aace417715c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.010257 449 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e] alter complete (version 2)
I20250629 01:51:55.011816 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ac560295fddd44368e6f9bc543f7cbfa
I20250629 01:51:55.012265 2217 raft_consensus.cc:491] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.012373 2217 raft_consensus.cc:513] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.012797 453 catalog_manager.cc:6652] ReplaceTablet: tablet ac560295fddd44368e6f9bc543f7cbfa deleted and replaced by tablet 9f6fbefddf434f32ab373b4e0191cee2
I20250629 01:51:55.012866 2217 leader_election.cc:290] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:55.013573 850 leader_election.cc:343] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0d925ffce89b40258b29ec021bfa5f92
I20250629 01:51:55.013759 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ac560295fddd44368e6f9bc543f7cbfa with delete_type TABLET_DATA_DELETED (replaced by tablet 9f6fbefddf434f32ab373b4e0191cee2) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.013957 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ac560295fddd44368e6f9bc543f7cbfa with delete_type TABLET_DATA_DELETED (replaced by tablet 9f6fbefddf434f32ab373b4e0191cee2) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:55.014473 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ac560295fddd44368e6f9bc543f7cbfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac560295fddd44368e6f9bc543f7cbfa already in progress: creating tablet
W20250629 01:51:55.014858 850 leader_election.cc:343] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0d925ffce89b40258b29ec021bfa5f92
I20250629 01:51:55.014945 850 leader_election.cc:304] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.015216 2217 raft_consensus.cc:2747] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.015831 1157 ts_tablet_manager.cc:1918] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.015698 453 catalog_manager.cc:5582] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
W20250629 01:51:55.016434 710 leader_election.cc:343] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e3b17172ab8b4e0dad36ebaac776dc03
I20250629 01:51:55.016530 710 leader_election.cc:304] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:55.018211 1154 tablet_bootstrap.cc:492] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.018347 1154 ts_tablet_manager.cc:1397] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20250629 01:51:55.018508 2228 raft_consensus.cc:2747] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.018674 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:55.018952 453 catalog_manager.cc:5582] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:55.019079 1154 raft_consensus.cc:357] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.019280 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]). Expected version 2 got 0
I20250629 01:51:55.019308 1154 raft_consensus.cc:383] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.019423 1154 raft_consensus.cc:738] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.019636 1154 consensus_queue.cc:260] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.020262 1154 ts_tablet_manager.cc:1428] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:55.020522 1154 tablet_bootstrap.cc:492] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.022486 1154 tablet_bootstrap.cc:654] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.025267 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_85e8350afe3f4983b2225325b3c58208"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.025696 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85e8350afe3f4983b2225325b3c58208 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.027715 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2cba6b4f427b44168c36647373065bd7" } new_table_name: "default.Table_8f1d82de01e44c02bbbf84faffeadb39" modify_external_catalogs: true
I20250629 01:51:55.028012 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]). Expected version 2 got 0
I20250629 01:51:55.028275 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_475fe7b17a5c4aac90faf6432e488d87 [id=0fb1390e9626454eba8e8d5c634e3943]). Expected version 4 got 0
I20250629 01:51:55.028445 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 0323947b135a4308903b1eed1ca4db71 (table default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]). Expected version 3 got 0
I20250629 01:51:55.028860 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e646217c593b412f8d30f34145ea5325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.029294 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e646217c593b412f8d30f34145ea5325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.030507 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_02190f4ee1a746e7b458b005042fbc0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.030875 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02190f4ee1a746e7b458b005042fbc0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.031946 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d1d0b27c4dae4594a84e9859774c89d6" } new_table_name: "default.Table_904b8b7a3ab1414a9bd29274648bbb86" modify_external_catalogs: true
I20250629 01:51:55.034550 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7e5e094e14304fd7b0899ba75536ea4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.034932 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e5e094e14304fd7b0899ba75536ea4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.036921 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:51:55.037628 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f5624ebd8475423285371a2a55453a30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.038014 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5624ebd8475423285371a2a55453a30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.030393 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ba9eabd06174b87a1485f4921895c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.040025 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_78723e0db17d407aaed94a045aac54c1" } new_table_name: "default.Table_c85076a59d9c4336942e15ebb642ed74" modify_external_catalogs: true
I20250629 01:51:55.040272 1861 raft_consensus.cc:491] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.040367 1861 raft_consensus.cc:513] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.040766 1861 leader_election.cc:290] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.043500 1158 tablet_replica.cc:331] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:55.043673 1158 raft_consensus.cc:2241] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.043781 1158 raft_consensus.cc:2270] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.044413 1158 ts_tablet_manager.cc:1905] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.046273 2238 tablet.cc:1722] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:55.046772 1155 tablet_bootstrap.cc:492] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.046861 1155 ts_tablet_manager.cc:1397] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.084s user 0.001s sys 0.002s
I20250629 01:51:55.047405 1155 raft_consensus.cc:357] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.047519 1155 raft_consensus.cc:383] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.047554 1155 raft_consensus.cc:738] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.047863 1155 consensus_queue.cc:260] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.048278 1157 log.cc:1199] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9cf553b291854a6f9e7ed78b5db5f18c
I20250629 01:51:55.048339 1155 ts_tablet_manager.cc:1428] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:55.048549 1155 tablet_bootstrap.cc:492] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.048717 1157 ts_tablet_manager.cc:1939] T 9cf553b291854a6f9e7ed78b5db5f18c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.049417 1157 tablet_replica.cc:331] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.049647 1157 raft_consensus.cc:2241] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.049737 1158 ts_tablet_manager.cc:1918] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.049866 1158 log.cc:1199] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0ba9eabd06174b87a1485f4921895c5f
I20250629 01:51:55.049923 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9cf553b291854a6f9e7ed78b5db5f18c (table default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e]) successfully deleted
I20250629 01:51:55.050104 1155 tablet_bootstrap.cc:654] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:55.050144 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 966e379adef8435887a2d8f39bb99968 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 966e379adef8435887a2d8f39bb99968 already in progress: creating tablet
W20250629 01:51:55.050330 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b already in progress: creating tablet
I20250629 01:51:55.050333 1158 ts_tablet_manager.cc:1939] T 0ba9eabd06174b87a1485f4921895c5f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:55.050503 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 989b26c3e2294005a1ebb4316c81ac29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 989b26c3e2294005a1ebb4316c81ac29 already in progress: creating tablet
I20250629 01:51:55.050769 2052 raft_consensus.cc:491] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.050869 2052 raft_consensus.cc:513] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.051295 2052 leader_election.cc:290] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.050668 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.052872 2183 raft_consensus.cc:491] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.053009 2183 raft_consensus.cc:513] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.053047 1048 raft_consensus.cc:2466] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:55.053849 2183 leader_election.cc:290] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.054103 1157 raft_consensus.cc:2270] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.054093 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38f831a6dd68493382d5f2920d0794a8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:55.054529 986 leader_election.cc:343] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.041612 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.055058 712 leader_election.cc:343] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.055589 1157 ts_tablet_manager.cc:1905] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.046077 888 tablet_service.cc:1468] Processing CreateTablet for tablet f6f82ee996c94196b912aace417715c9 (DEFAULT_TABLE table=default.Table_d17573a245c247849aa573b35b93e377 [id=cb20a9a97250475bb854c469ee22f8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.047969 1159 raft_consensus.cc:383] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.020921 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ac560295fddd44368e6f9bc543f7cbfa with delete_type TABLET_DATA_DELETED (replaced by tablet 9f6fbefddf434f32ab373b4e0191cee2) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.051950 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bbbeb43473a4f6f892056a30f62589f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.054323 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38f831a6dd68493382d5f2920d0794a8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.056542 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f82ee996c94196b912aace417715c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.056806 988 leader_election.cc:343] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:55.056842 712 leader_election.cc:343] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.056913 988 leader_election.cc:304] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.057049 1159 raft_consensus.cc:738] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.057183 2183 raft_consensus.cc:2747] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.057317 1159 consensus_queue.cc:260] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.057832 1159 ts_tablet_manager.cc:1428] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.122s user 0.015s sys 0.003s
I20250629 01:51:55.058182 1159 tablet_bootstrap.cc:492] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.059330 1159 tablet_bootstrap.cc:654] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.061679 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 47d4287317ec4788bd96ad174c31748d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.067018 1159 tablet_bootstrap.cc:492] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.068881 2271 tablet.cc:1722] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:55.069537 1159 ts_tablet_manager.cc:1397] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20250629 01:51:55.071944 1154 tablet_bootstrap.cc:492] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.072079 1154 ts_tablet_manager.cc:1397] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.001s
I20250629 01:51:55.072635 1154 raft_consensus.cc:357] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.072813 1154 raft_consensus.cc:383] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.072888 1154 raft_consensus.cc:738] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.073081 1154 consensus_queue.cc:260] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.073500 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]). Expected version 2 got 0
I20250629 01:51:55.076730 1154 ts_tablet_manager.cc:1428] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:55.077001 1154 tablet_bootstrap.cc:492] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.078390 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:55.079587 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9914b6019e564cf7aa3ad9c511898fed" } new_table_name: "default.Table_fcb861de87ad468584f2f0c88b9db3b6" modify_external_catalogs: true
I20250629 01:51:55.081450 1154 tablet_bootstrap.cc:654] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.081991 1159 raft_consensus.cc:357] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.082131 1159 raft_consensus.cc:383] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.082180 1159 raft_consensus.cc:738] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.082331 1159 consensus_queue.cc:260] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.088456 1159 ts_tablet_manager.cc:1428] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.019s user 0.000s sys 0.001s
I20250629 01:51:55.089593 1157 ts_tablet_manager.cc:1918] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.089912 1157 log.cc:1199] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/df66984acfa44a99bf69096aff6d49ab
I20250629 01:51:55.090205 1157 ts_tablet_manager.cc:1939] T df66984acfa44a99bf69096aff6d49ab P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.089833 1159 tablet_bootstrap.cc:492] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.091575 2052 raft_consensus.cc:491] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.091694 2052 raft_consensus.cc:513] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.092195 2052 leader_election.cc:290] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.093489 2217 raft_consensus.cc:491] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.093596 2217 raft_consensus.cc:513] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.094039 2217 leader_election.cc:290] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:55.094858 850 leader_election.cc:343] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:55.095142 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.095446 712 leader_election.cc:343] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.031174 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6bf3c7d0bb548e0b77ba8b569a72d0d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.096038 850 leader_election.cc:343] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:55.096202 850 leader_election.cc:304] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.096664 1159 tablet_bootstrap.cc:654] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.096797 2217 raft_consensus.cc:2747] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:55.097239 712 leader_election.cc:343] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.097983 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet df66984acfa44a99bf69096aff6d49ab (table default.Table_abc4bc33793d4bee824f97f24a172593 [id=8a6a13c4f09547aa8ca713ab9892f557]) successfully deleted
W20250629 01:51:55.098663 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 31cee0e741584157b0333851f94c8559 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31cee0e741584157b0333851f94c8559 already in progress: creating tablet
I20250629 01:51:55.099311 1157 tablet_replica.cc:331] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.099740 1157 raft_consensus.cc:2241] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.100188 1157 raft_consensus.cc:2270] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.101397 1157 ts_tablet_manager.cc:1905] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.105191 1159 tablet_bootstrap.cc:492] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.105329 1159 ts_tablet_manager.cc:1397] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20250629 01:51:55.105988 1159 raft_consensus.cc:357] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.107527 1159 raft_consensus.cc:383] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.107694 1159 raft_consensus.cc:738] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.107959 1159 consensus_queue.cc:260] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.111021 1159 ts_tablet_manager.cc:1428] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250629 01:51:55.111124 1155 tablet_bootstrap.cc:492] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.111595 1159 tablet_bootstrap.cc:492] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.113576 1155 ts_tablet_manager.cc:1397] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.008s
I20250629 01:51:55.116724 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:55.117058 1159 tablet_bootstrap.cc:654] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:55.118502 712 consensus_peers.cc:489] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:55.119215 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_51f8fff3136f4bdeb43af63ea621a31e" } modify_external_catalogs: true
I20250629 01:51:55.118870 1155 raft_consensus.cc:357] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.119414 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_51f8fff3136f4bdeb43af63ea621a31e" } modify_external_catalogs: true
I20250629 01:51:55.119704 1155 raft_consensus.cc:383] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.120289 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6d95a42070914b1b95a20fb55e589351" } new_table_name: "default.Table_cc8b682489c3440583a2b90e20fe6e79" modify_external_catalogs: true
I20250629 01:51:55.120131 1155 raft_consensus.cc:738] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
W20250629 01:51:55.120749 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e62e0683ffd4fdb80cd9646ddcc6692 consensus state has no leader
I20250629 01:51:55.121043 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97" } modify_external_catalogs: true
I20250629 01:51:55.121206 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d6cfea9bf4764b5a8fd3bf29e0b47a97" } modify_external_catalogs: true
I20250629 01:51:55.121696 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4356a70d10bd41e4ad6b77e5612a81ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.122124 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4356a70d10bd41e4ad6b77e5612a81ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.121742 1155 consensus_queue.cc:260] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.122566 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_966c88ce78e148fe9366b092be734d02" } modify_external_catalogs: true
I20250629 01:51:55.122694 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_966c88ce78e148fe9366b092be734d02" } modify_external_catalogs: true
W20250629 01:51:55.122943 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5e8b8e4d42f43c78f34102b76b4b11a consensus state has no leader
I20250629 01:51:55.123157 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_17373cbe989b4dfcb4c39ed0f708d216" } modify_external_catalogs: true
I20250629 01:51:55.123268 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_17373cbe989b4dfcb4c39ed0f708d216" } modify_external_catalogs: true
I20250629 01:51:55.123636 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9114ba5320214ff6bb77aebc1d32ce9e" } modify_external_catalogs: true
I20250629 01:51:55.123761 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9114ba5320214ff6bb77aebc1d32ce9e" } modify_external_catalogs: true
I20250629 01:51:55.124161 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5257d4d9f670497287003f5c8bf98456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.124563 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5257d4d9f670497287003f5c8bf98456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:55.124858 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf676660c9b44ffd87e365287e6fc0a9 consensus state has no leader
W20250629 01:51:55.125159 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 303f77a729c648f9b1146d11056b242d consensus state has no leader
I20250629 01:51:55.125241 1157 ts_tablet_manager.cc:1918] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.125313 1157 log.cc:1199] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4a692f00ba244a80b9a6d07029f8d27e
I20250629 01:51:55.125228 1155 ts_tablet_manager.cc:1428] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250629 01:51:55.125540 1157 ts_tablet_manager.cc:1939] T 4a692f00ba244a80b9a6d07029f8d27e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.127058 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 683c87ea2ad44cecb2366d4a846d9195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.128063 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4a692f00ba244a80b9a6d07029f8d27e (table default.Table_e605f1f30c8d4fdd8503b71227359798 [id=285208f892d84eea88ede0f96db5e7cf]) successfully deleted
I20250629 01:51:55.128298 740 tablet_service.cc:1515] Processing DeleteTablet for tablet da93c730451d42bc8df1e582fb3af818 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.128685 1157 tablet_replica.cc:331] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.129132 1157 raft_consensus.cc:2241] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.129586 1157 raft_consensus.cc:2270] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.131243 1155 tablet_bootstrap.cc:492] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.133594 1157 ts_tablet_manager.cc:1905] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.134352 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 943f584fb9554fa4b65e5302443cec92 (table default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e]). Expected version 1 got 0
I20250629 01:51:55.143013 1155 tablet_bootstrap.cc:654] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.145185 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed] alter complete (version 2)
I20250629 01:51:55.146358 1159 tablet_bootstrap.cc:492] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.146742 1159 ts_tablet_manager.cc:1397] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.004s
I20250629 01:51:55.146746 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 2aca26af31ed4bc994f4d62603edc881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.146979 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c995ce14c9a4966b842fcba520cf87d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.147863 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ee1baa976b5e47e490a0fd561b37f83d
I20250629 01:51:55.149077 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet caa63fe10ff9487a9fc16479285c5abd
I20250629 01:51:55.148896 888 tablet_service.cc:1515] Processing DeleteTablet for tablet ee1baa976b5e47e490a0fd561b37f83d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.150591 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8cea647451954d8ab3bd495a3a080c97
I20250629 01:51:55.150604 1159 raft_consensus.cc:357] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.150727 1159 raft_consensus.cc:383] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.150715 740 tablet_service.cc:1515] Processing DeleteTablet for tablet ee1baa976b5e47e490a0fd561b37f83d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.150764 1159 raft_consensus.cc:738] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.150893 749 tablet_service.cc:1515] Processing DeleteTablet for tablet caa63fe10ff9487a9fc16479285c5abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.149999 895 tablet_service.cc:1515] Processing DeleteTablet for tablet caa63fe10ff9487a9fc16479285c5abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.151310 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cea647451954d8ab3bd495a3a080c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.151654 1159 consensus_queue.cc:260] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.151717 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cea647451954d8ab3bd495a3a080c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:55.152276 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa2052970beb4e769b61d7cd3edc0db6 consensus state has no leader
I20250629 01:51:55.152501 1159 ts_tablet_manager.cc:1428] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:55.152776 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 06d591c66d734f76a1bda2194ec62776
I20250629 01:51:55.153179 1159 tablet_bootstrap.cc:492] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.153829 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d591c66d734f76a1bda2194ec62776 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.154481 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d591c66d734f76a1bda2194ec62776 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.155368 1159 tablet_bootstrap.cc:654] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.156162 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 541728ff8dc3471ba90784b4a769c8e1 (table default.Table_a5c12815f8f24b89aa5d482cde53223c [id=fd0ea1adec3e487ab647680c75b7fde8]). Expected version 1 got 0
I20250629 01:51:55.150060 1157 ts_tablet_manager.cc:1918] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.156733 1157 log.cc:1199] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/81b32bac5a2144d1943285f5f446d53d
I20250629 01:51:55.157127 1157 ts_tablet_manager.cc:1939] T 81b32bac5a2144d1943285f5f446d53d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.157366 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 15863974418e4fb5a0447c75ccdc2e38
I20250629 01:51:55.158860 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 81b32bac5a2144d1943285f5f446d53d (table default.Table_4559cd80e6874b7fb3e1ea565b90fd03 [id=985a6813b1804db2b8df121cf1f31553]) successfully deleted
I20250629 01:51:55.159988 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 541728ff8dc3471ba90784b4a769c8e1 (table default.Table_a5c12815f8f24b89aa5d482cde53223c [id=fd0ea1adec3e487ab647680c75b7fde8]). Expected version 1 got 0
W20250629 01:51:55.160236 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f4cf919a96f34167b7dc9c3dfe7946a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4cf919a96f34167b7dc9c3dfe7946a4 already in progress: creating tablet
I20250629 01:51:55.160790 1157 tablet_replica.cc:331] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:55.161033 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cfb025535ecd4cf88681c4951a56ba29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfb025535ecd4cf88681c4951a56ba29 already in progress: creating tablet
I20250629 01:51:55.166466 1155 tablet_bootstrap.cc:492] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.166725 1157 raft_consensus.cc:2241] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.167095 1157 raft_consensus.cc:2270] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.161753 1159 tablet_bootstrap.cc:492] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.167825 1155 ts_tablet_manager.cc:1397] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.004s
I20250629 01:51:55.167956 1159 ts_tablet_manager.cc:1397] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20250629 01:51:55.168563 1159 raft_consensus.cc:357] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.168730 1159 raft_consensus.cc:383] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.168819 1159 raft_consensus.cc:738] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.168855 1155 raft_consensus.cc:357] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.169121 1155 raft_consensus.cc:383] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.169039 1159 consensus_queue.cc:260] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.169237 1155 raft_consensus.cc:738] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.169723 1159 ts_tablet_manager.cc:1428] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:55.170073 1159 tablet_bootstrap.cc:492] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.170676 1155 consensus_queue.cc:260] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.171370 1155 ts_tablet_manager.cc:1428] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250629 01:51:55.173816 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 75ce1bd4af434f7e99092aafc0ab0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.174831 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:55.175375 1155 tablet_bootstrap.cc:492] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.176194 1159 tablet_bootstrap.cc:654] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.176575 1157 ts_tablet_manager.cc:1905] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.179224 916 raft_consensus.cc:3058] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.179561 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
W20250629 01:51:55.183259 2233 tablet_replica.cc:1408] Time spent flushing the Log queue took a long time: real 0.285s user 0.000s sys 0.000s
I20250629 01:51:55.189193 749 tablet_service.cc:1468] Processing CreateTablet for tablet 0eaa38e09c874844aa2320a255684a3a (DEFAULT_TABLE table=default.Table_2c05f26e7f594408b830d3c545b58966 [id=2ed86e3d11d7479385ab44c255144265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.189536 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eaa38e09c874844aa2320a255684a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.189558 895 tablet_service.cc:1468] Processing CreateTablet for tablet f1ad1b7de79b48058437a9dea5d30b37 (DEFAULT_TABLE table=default.Table_97c5093f3a1542f5bb7fc463c1955f9e [id=6faf9bc8cce84174a48928674dc25702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.189838 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ad1b7de79b48058437a9dea5d30b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.195253 740 tablet_service.cc:1468] Processing CreateTablet for tablet f1ad1b7de79b48058437a9dea5d30b37 (DEFAULT_TABLE table=default.Table_97c5093f3a1542f5bb7fc463c1955f9e [id=6faf9bc8cce84174a48928674dc25702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.195554 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ad1b7de79b48058437a9dea5d30b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.196915 1156 ts_tablet_manager.cc:1918] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.197103 1156 log.cc:1199] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5c6f3f19a4d441f58c8c6099e04f82ec
I20250629 01:51:55.197438 1156 ts_tablet_manager.cc:1939] T 5c6f3f19a4d441f58c8c6099e04f82ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:55.201239 1157 ts_tablet_manager.cc:1918] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:55.202832 895 tablet_service.cc:1468] Processing CreateTablet for tablet c4530a43bdeb47d3ad29b17750be12d2 (DEFAULT_TABLE table=default.Table_7cf9a31f57704fea83af6665be37020e [id=07f8379f2af6486995817a514e02130a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.203708 1157 log.cc:1199] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/798516e55ca747fcb2942026adb78af0
I20250629 01:51:55.203855 888 tablet_service.cc:1468] Processing CreateTablet for tablet 7cefc4de48e844f496459d7ae966ab52 (DEFAULT_TABLE table=default.Table_0814dc6691b040e1a9f69f258973cf28 [id=896fee5246524ebabd2f166ae669fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.204249 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e4309edc8dd64fc88947c246a80faac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.204392 1157 ts_tablet_manager.cc:1939] T 798516e55ca747fcb2942026adb78af0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:55.204828 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4309edc8dd64fc88947c246a80faac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.207181 745 tablet_service.cc:1468] Processing CreateTablet for tablet c4530a43bdeb47d3ad29b17750be12d2 (DEFAULT_TABLE table=default.Table_7cf9a31f57704fea83af6665be37020e [id=07f8379f2af6486995817a514e02130a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.207520 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4530a43bdeb47d3ad29b17750be12d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.208045 738 tablet_service.cc:1468] Processing CreateTablet for tablet 7cefc4de48e844f496459d7ae966ab52 (DEFAULT_TABLE table=default.Table_0814dc6691b040e1a9f69f258973cf28 [id=896fee5246524ebabd2f166ae669fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.208422 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cefc4de48e844f496459d7ae966ab52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.211174 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_45ad1da33f23408c858cd346e7ac71bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.211577 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45ad1da33f23408c858cd346e7ac71bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.213963 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cd351e2fabea4665b7954d5eca538f02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.214380 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd351e2fabea4665b7954d5eca538f02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.214789 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cefc4de48e844f496459d7ae966ab52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.215137 1157 tablet_replica.cc:331] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.215263 1157 raft_consensus.cc:2241] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.215268 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]) successfully deleted
I20250629 01:51:55.215408 1157 raft_consensus.cc:2270] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.215818 1157 ts_tablet_manager.cc:1905] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.219745 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4530a43bdeb47d3ad29b17750be12d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.228111 888 tablet_service.cc:1468] Processing CreateTablet for tablet 9479685385b5402b9ec21249621b1600 (DEFAULT_TABLE table=default.Table_eee311f8e86f4b1da839f4f16aa2b343 [id=ff22b7d417a741559c91ef3360e0d50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.228427 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9479685385b5402b9ec21249621b1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.228508 890 tablet_service.cc:1468] Processing CreateTablet for tablet 72f277d4234744a2995953a5eb1715b3 (DEFAULT_TABLE table=default.Table_b5a50c642fa1422bb612d0edeadb4ee7 [id=e3d515b8692445e5afdea63aeadedc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.228824 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f277d4234744a2995953a5eb1715b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.229640 881 tablet_service.cc:1468] Processing CreateTablet for tablet 30265f664bff4b058e587edd9e0c5978 (DEFAULT_TABLE table=default.Table_9a012ad660f649da977b3123b27ecd2e [id=43c1739e8c674274a89bbfc16f16e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.229929 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30265f664bff4b058e587edd9e0c5978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.230373 750 tablet_service.cc:1468] Processing CreateTablet for tablet 30265f664bff4b058e587edd9e0c5978 (DEFAULT_TABLE table=default.Table_9a012ad660f649da977b3123b27ecd2e [id=43c1739e8c674274a89bbfc16f16e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.230655 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30265f664bff4b058e587edd9e0c5978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.231407 1157 ts_tablet_manager.cc:1918] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.232422 1157 log.cc:1199] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/432680cae61946af9859760fe23110ec
I20250629 01:51:55.232882 1157 ts_tablet_manager.cc:1939] T 432680cae61946af9859760fe23110ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.233691 740 tablet_service.cc:1468] Processing CreateTablet for tablet 72f277d4234744a2995953a5eb1715b3 (DEFAULT_TABLE table=default.Table_b5a50c642fa1422bb612d0edeadb4ee7 [id=e3d515b8692445e5afdea63aeadedc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.233875 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 432680cae61946af9859760fe23110ec (table default.Table_c70f3648ce00471bb5d6dbd7d12e6b50 [id=2dfa777b5eab4acb9adfd35ab56c17d0]) successfully deleted
I20250629 01:51:55.234102 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f277d4234744a2995953a5eb1715b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.234594 1157 tablet_replica.cc:331] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.235551 738 tablet_service.cc:1468] Processing CreateTablet for tablet 9479685385b5402b9ec21249621b1600 (DEFAULT_TABLE table=default.Table_eee311f8e86f4b1da839f4f16aa2b343 [id=ff22b7d417a741559c91ef3360e0d50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.235966 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9479685385b5402b9ec21249621b1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.236028 743 tablet_service.cc:1468] Processing CreateTablet for tablet c1b5deb7f2204e3aa271ddd818a360d3 (DEFAULT_TABLE table=default.Table_5d60769b6bad4b8eb8dfa90c3047c054 [id=777823b736a74d3585be9f11f0ffe055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.236384 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b5deb7f2204e3aa271ddd818a360d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.247234 1157 raft_consensus.cc:2241] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.247757 1157 raft_consensus.cc:2270] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.248256 1157 ts_tablet_manager.cc:1905] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.253082 1157 ts_tablet_manager.cc:1918] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.257609 1155 tablet_bootstrap.cc:654] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.259104 1157 log.cc:1199] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/34172d05ae40417494cd8f67f4cb981e
I20250629 01:51:55.262713 1157 ts_tablet_manager.cc:1939] T 34172d05ae40417494cd8f67f4cb981e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.263207 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 34172d05ae40417494cd8f67f4cb981e (table default.Table_55b080a79b3540e6bcf0cf0b5f6fcbf0 [id=3ab9d677112a45308d91465d1ee19c62]) successfully deleted
I20250629 01:51:55.263782 1157 tablet_replica.cc:331] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.264598 1157 raft_consensus.cc:2241] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.264917 1157 raft_consensus.cc:2270] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.265654 1157 ts_tablet_manager.cc:1905] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.269320 1155 tablet_bootstrap.cc:492] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.270175 1155 ts_tablet_manager.cc:1397] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.002s
I20250629 01:51:55.270332 895 tablet_service.cc:1468] Processing CreateTablet for tablet c1b5deb7f2204e3aa271ddd818a360d3 (DEFAULT_TABLE table=default.Table_5d60769b6bad4b8eb8dfa90c3047c054 [id=777823b736a74d3585be9f11f0ffe055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.270704 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b5deb7f2204e3aa271ddd818a360d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.270898 1155 raft_consensus.cc:357] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.271171 1155 raft_consensus.cc:383] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.271291 1155 raft_consensus.cc:738] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.271559 1155 consensus_queue.cc:260] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.272387 1155 ts_tablet_manager.cc:1428] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:55.274299 1155 tablet_bootstrap.cc:492] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.274811 1159 tablet_bootstrap.cc:492] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.274964 1159 ts_tablet_manager.cc:1397] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.105s user 0.000s sys 0.021s
I20250629 01:51:55.275645 1159 raft_consensus.cc:357] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.277374 1155 tablet_bootstrap.cc:654] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.277740 1159 raft_consensus.cc:383] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.277909 1159 raft_consensus.cc:738] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.278183 1159 consensus_queue.cc:260] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.281301 1159 ts_tablet_manager.cc:1428] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:51:55.281852 1159 tablet_bootstrap.cc:492] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.280112 2052 raft_consensus.cc:491] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.285141 2052 raft_consensus.cc:513] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.285684 2052 leader_election.cc:290] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.286223 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.286689 712 leader_election.cc:343] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.287825 1159 tablet_bootstrap.cc:654] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.288404 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 984) took 1192 ms. Trace:
I20250629 01:51:55.288579 1156 rpcz_store.cc:276] 0629 01:51:54.096019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.096335 (+ 316us) service_pool.cc:225] Handling call
0629 01:51:54.899630 (+803295us) ts_tablet_manager.cc:1145] Deleting tablet 5c6f3f19a4d441f58c8c6099e04f82ec
0629 01:51:54.899640 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:54.911995 (+ 12355us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.196811 (+284816us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.288244 (+ 91433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":800433}
I20250629 01:51:55.289089 1155 tablet_bootstrap.cc:492] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.289070 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5c6f3f19a4d441f58c8c6099e04f82ec (table default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]) successfully deleted
I20250629 01:51:55.292490 1155 ts_tablet_manager.cc:1397] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250629 01:51:55.293453 1154 tablet_bootstrap.cc:492] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.293619 1154 ts_tablet_manager.cc:1397] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.217s user 0.013s sys 0.026s
I20250629 01:51:55.293769 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0ba9eabd06174b87a1485f4921895c5f (table default.Table_6e4d952c729c4984a21f16da686e28f0 [id=2a01904ec4bd4b48b528d7c0cfca4a43]) successfully deleted
I20250629 01:51:55.294291 1154 raft_consensus.cc:357] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.294485 1154 raft_consensus.cc:383] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.294569 1154 raft_consensus.cc:738] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.294829 1154 consensus_queue.cc:260] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.295285 1154 ts_tablet_manager.cc:1428] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:55.295820 2183 raft_consensus.cc:491] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.295928 2183 raft_consensus.cc:513] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.296396 2183 leader_election.cc:290] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.297287 1154 tablet_bootstrap.cc:492] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.298331 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ee1baa976b5e47e490a0fd561b37f83d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.298311 1155 raft_consensus.cc:357] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.298465 1155 raft_consensus.cc:383] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.297042 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1e22a0c02054c7db514eefe20ee3311" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.299592 988 leader_election.cc:343] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.299993 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet caa63fe10ff9487a9fc16479285c5abd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.301134 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8cea647451954d8ab3bd495a3a080c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.293540 2238 tablet.cc:1722] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:55.301214 1155 raft_consensus.cc:738] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.296873 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1e22a0c02054c7db514eefe20ee3311" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:55.289492 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 987) took 1174 ms. Trace:
I20250629 01:51:55.294409 710 leader_election.cc:304] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.301440 775 raft_consensus.cc:2466] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:55.301786 2049 raft_consensus.cc:2802] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:55.301796 986 leader_election.cc:304] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.301887 2049 raft_consensus.cc:491] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:55.301972 2049 raft_consensus.cc:3058] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.302850 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 06d591c66d734f76a1bda2194ec62776 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.303886 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0eaa38e09c874844aa2320a255684a3a (DEFAULT_TABLE table=default.Table_2c05f26e7f594408b830d3c545b58966 [id=2ed86e3d11d7479385ab44c255144265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.304200 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eaa38e09c874844aa2320a255684a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.304800 2049 raft_consensus.cc:513] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.304869 2273 raft_consensus.cc:2802] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:55.304931 2273 raft_consensus.cc:491] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:55.304981 2273 raft_consensus.cc:3058] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.305583 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:55.305824 2049 leader_election.cc:290] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:55.306113 712 leader_election.cc:343] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.306561 1156 rpcz_store.cc:276] 0629 01:51:54.114584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.117247 (+ 2663us) service_pool.cc:225] Handling call
0629 01:51:55.289363 (+1172116us) ts_tablet_manager.cc:1145] Deleting tablet 966e379adef8435887a2d8f39bb99968
0629 01:51:55.289373 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.289457 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171988}
W20250629 01:51:55.307027 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 966e379adef8435887a2d8f39bb99968 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 966e379adef8435887a2d8f39bb99968 already in progress: creating tablet
I20250629 01:51:55.307250 2273 raft_consensus.cc:513] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.307541 1155 consensus_queue.cc:260] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.308138 2273 leader_election.cc:290] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.308189 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1e22a0c02054c7db514eefe20ee3311" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:55.284840 1157 ts_tablet_manager.cc:1918] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:55.308555 988 leader_election.cc:343] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.308597 1157 log.cc:1199] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a485e0e6e2f04aa6bd67a62f6d30be26
I20250629 01:51:55.309291 1157 ts_tablet_manager.cc:1939] T a485e0e6e2f04aa6bd67a62f6d30be26 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.309592 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1e22a0c02054c7db514eefe20ee3311" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:55.309731 1155 ts_tablet_manager.cc:1428] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:55.309767 775 raft_consensus.cc:3058] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.310118 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a485e0e6e2f04aa6bd67a62f6d30be26 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:51:55.310356 1157 tablet_replica.cc:331] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:51:55.310636 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 109e19ed0ea64fa4985e93efbaca5107 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 109e19ed0ea64fa4985e93efbaca5107 already in progress: creating tablet
W20250629 01:51:55.310820 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8992da4b777d400b9c3f242444229353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8992da4b777d400b9c3f242444229353 already in progress: creating tablet
W20250629 01:51:55.310990 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5614713dab134cb9b6d70c9a53973014 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5614713dab134cb9b6d70c9a53973014 already in progress: creating tablet
I20250629 01:51:55.312032 775 raft_consensus.cc:2466] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:55.312470 986 leader_election.cc:304] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.314286 1157 raft_consensus.cc:2241] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.314453 1157 raft_consensus.cc:2270] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.315201 1157 ts_tablet_manager.cc:1905] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.317869 1154 tablet_bootstrap.cc:654] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.318235 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 52be184d0480495a894442d6f72a0f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.318365 1159 tablet_bootstrap.cc:492] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.318506 1159 ts_tablet_manager.cc:1397] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.002s
I20250629 01:51:55.319140 1159 raft_consensus.cc:357] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.319325 1159 raft_consensus.cc:383] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.319408 1159 raft_consensus.cc:738] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.319630 1159 consensus_queue.cc:260] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.320159 1159 ts_tablet_manager.cc:1428] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:55.320410 1159 tablet_bootstrap.cc:492] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.321758 1159 tablet_bootstrap.cc:654] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.325033 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 988) took 1210 ms. Trace:
I20250629 01:51:55.325150 1156 rpcz_store.cc:276] 0629 01:51:54.114669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.121133 (+ 6464us) service_pool.cc:225] Handling call
0629 01:51:55.307311 (+1186178us) ts_tablet_manager.cc:1145] Deleting tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b
0629 01:51:55.324902 (+ 17591us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.324996 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1186016}
W20250629 01:51:55.325654 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7faabb2f5d4c40cdb4b8ba5fb7ada92b already in progress: creating tablet
I20250629 01:51:55.326689 2283 tablet.cc:1722] T 541728ff8dc3471ba90784b4a769c8e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:55.327840 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2a7ba606f3fe4a2b8233338d373dd389"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.328362 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a7ba606f3fe4a2b8233338d373dd389 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.328548 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7e0ccdf1ddb349558670c1d0a61d5cf2" } new_table_name: "default.Table_b4abe11363964e61a884d8bf8f8d0189" modify_external_catalogs: true
I20250629 01:51:55.329432 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c3c28291b8e046aebf0a8accc24e0599"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.329814 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3c28291b8e046aebf0a8accc24e0599 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.331980 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2657211e01984ab381578b3c21d851ea" } new_table_name: "default.Table_8876065bf7414680b4cac90521c176d3" modify_external_catalogs: true
I20250629 01:51:55.332129 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_899f5cb3f36f46dd8e4c809803070720" } new_table_name: "default.Table_f27590f1ad614dafa2cb4866df1b83d0" modify_external_catalogs: true
I20250629 01:51:55.333250 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_db4b4ae9a38f4d75845beeffd7251609"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.333449 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 541728ff8dc3471ba90784b4a769c8e1 (table default.Table_a5c12815f8f24b89aa5d482cde53223c [id=fd0ea1adec3e487ab647680c75b7fde8]). Expected version 1 got 0
W20250629 01:51:55.333751 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db4b4ae9a38f4d75845beeffd7251609 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.334156 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3cd6904b29dc4d52bb6acb613babf904"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.334517 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cd6904b29dc4d52bb6acb613babf904 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.334911 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_02f34269d06f4dfb9d03d3da0aea953d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.335345 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02f34269d06f4dfb9d03d3da0aea953d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.335551 1157 ts_tablet_manager.cc:1918] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.335623 1157 log.cc:1199] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0d925ffce89b40258b29ec021bfa5f92
I20250629 01:51:55.335887 1157 ts_tablet_manager.cc:1939] T 0d925ffce89b40258b29ec021bfa5f92 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.336342 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0d925ffce89b40258b29ec021bfa5f92 (table default.Table_1b37a535cd2f483a8c72aa10882a4301 [id=b516da4148384818bcc69e4259f20ef9]) successfully deleted
W20250629 01:51:55.336854 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9e0d550773c949df9992be1f9f01b548 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e0d550773c949df9992be1f9f01b548 already in progress: creating tablet
I20250629 01:51:55.337167 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8a2b5b69e5db47f89ce6aafa5f16bd59" } new_table_name: "default.Table_0a878b9a2af948319e9adaf6783959a0" modify_external_catalogs: true
I20250629 01:51:55.341624 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:55.342810 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 196f1bd233ab4399a9d22a55b8bc2635 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:55.342974 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46c71d199da74227bd115c3a94178512 consensus state has no leader
W20250629 01:51:55.343706 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac2270c5abd4456890378c7f82b5f663 consensus state has no leader
I20250629 01:51:55.345638 1157 tablet_replica.cc:331] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.346251 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 993) took 1195 ms. Trace:
I20250629 01:51:55.346354 1156 rpcz_store.cc:276] 0629 01:51:54.150859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.166499 (+ 15640us) service_pool.cc:225] Handling call
0629 01:51:55.325373 (+1158874us) ts_tablet_manager.cc:1145] Deleting tablet 989b26c3e2294005a1ebb4316c81ac29
0629 01:51:55.346132 (+ 20759us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.346212 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1158659}
I20250629 01:51:55.346642 1156 tablet_replica.cc:331] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:55.346840 1156 raft_consensus.cc:2241] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:55.346884 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 989b26c3e2294005a1ebb4316c81ac29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 989b26c3e2294005a1ebb4316c81ac29 already in progress: creating tablet
I20250629 01:51:55.346971 1156 raft_consensus.cc:2270] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.347427 1156 ts_tablet_manager.cc:1905] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.349787 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f9e7762f7fba4fc59cd1dd954b009b6b" } modify_external_catalogs: true
W20250629 01:51:55.349884 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0eeb0ef02c416abc5665f058c6bd7f consensus state has no leader
I20250629 01:51:55.349941 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f9e7762f7fba4fc59cd1dd954b009b6b" } modify_external_catalogs: true
I20250629 01:51:55.352141 2049 raft_consensus.cc:491] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.352269 2049 raft_consensus.cc:513] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.352222 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0fc926b866e240e592ab2c1505f115f5" } modify_external_catalogs: true
I20250629 01:51:55.352761 2049 leader_election.cc:290] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.352738 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0fc926b866e240e592ab2c1505f115f5" } modify_external_catalogs: true
I20250629 01:51:55.352516 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c5ad96e3c69d4a29ab6c662b82774b53" } modify_external_catalogs: true
I20250629 01:51:55.353010 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a08dd1de5190465b9b899ba5eaf80fff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.353121 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c5ad96e3c69d4a29ab6c662b82774b53" } modify_external_catalogs: true
W20250629 01:51:55.353436 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a08dd1de5190465b9b899ba5eaf80fff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.353981 1155 tablet_bootstrap.cc:492] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.355280 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fcb861de87ad468584f2f0c88b9db3b6" } modify_external_catalogs: true
I20250629 01:51:55.355428 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fcb861de87ad468584f2f0c88b9db3b6" } modify_external_catalogs: true
I20250629 01:51:55.355664 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5326881cf36c482ba24c3b952cf65c60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.356179 1155 tablet_bootstrap.cc:654] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:55.356194 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5326881cf36c482ba24c3b952cf65c60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.357815 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5829573f0cc6418ab66487ff0902cab5" } new_table_name: "default.Table_7de9393b563e46a28e254714961f0d2e" modify_external_catalogs: true
I20250629 01:51:55.358743 1156 ts_tablet_manager.cc:1918] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.358901 1156 log.cc:1199] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:55.358945 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f3aaa9e9eb034a5cbbd31b1529d0080a" } modify_external_catalogs: true
I20250629 01:51:55.359107 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f3aaa9e9eb034a5cbbd31b1529d0080a" } modify_external_catalogs: true
I20250629 01:51:55.359360 1156 ts_tablet_manager.cc:1939] T cabb0298771e4fd5be29e530d3a8ed3d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:55.359967 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 997) took 1175 ms. Trace:
I20250629 01:51:55.360054 1156 rpcz_store.cc:276] 0629 01:51:54.184550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.184604 (+ 54us) service_pool.cc:225] Handling call
0629 01:51:55.346586 (+1161982us) ts_tablet_manager.cc:1145] Deleting tablet cabb0298771e4fd5be29e530d3a8ed3d
0629 01:51:55.346593 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.349993 (+ 3400us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.358580 (+ 8587us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.359899 (+ 1319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161817}
I20250629 01:51:55.360518 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet cabb0298771e4fd5be29e530d3a8ed3d (table default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]) successfully deleted
I20250629 01:51:55.360926 1156 tablet_replica.cc:331] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:55.361140 1156 raft_consensus.cc:2241] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.362709 1156 raft_consensus.cc:2270] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.363430 1156 ts_tablet_manager.cc:1905] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.368314 738 tablet_service.cc:1468] Processing CreateTablet for tablet c56b761dc3b74621ac786ac1d786d539 (DEFAULT_TABLE table=default.Table_c4f60a9a65f54ed781a9b1643ba15c34 [id=397a102cc27743f18ddf17277d5b10b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.368675 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56b761dc3b74621ac786ac1d786d539. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.372920 740 tablet_service.cc:1468] Processing CreateTablet for tablet 969cffe4ef854409a3853a1c1d0d9a9a (DEFAULT_TABLE table=default.Table_8ea631c388eb4b60923f367cd108d236 [id=15079533f87d460f97e5ae86792b35eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.373035 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba02adec0d74547ab12dff3a0c898f7 (DEFAULT_TABLE table=default.Table_36921981749241789ba5b79d109b0549 [id=48e4102eef824437b95a71f60c5c1ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.373077 750 tablet_service.cc:1468] Processing CreateTablet for tablet 1a8dddc0659a4340af7f2d01c9a3151f (DEFAULT_TABLE table=default.Table_aa4b088ab043449280d641b0ff03d845 [id=aa724b93d430458095befa5f20b1ef70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.373220 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969cffe4ef854409a3853a1c1d0d9a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.375415 755 tablet_service.cc:1468] Processing CreateTablet for tablet 13d779335d09442682554322b59f915a (DEFAULT_TABLE table=default.Table_2698ae66abbc4163beb2f3939554be2a [id=bce6f6d30f594035b37966090fcf9d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.353444 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c995ce14c9a4966b842fcba520cf87d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.366142 888 tablet_service.cc:1468] Processing CreateTablet for tablet 1ecf731c6cfa4656acc992e2f8e06c08 (DEFAULT_TABLE table=default.Table_6a6e2363eabb4ce18ec597b680b149d5 [id=94c53c71b4c140dbaa88bb9984a26e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.377652 1157 raft_consensus.cc:2241] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.377878 1157 raft_consensus.cc:2270] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:55.377859 712 leader_election.cc:343] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.367187 881 tablet_service.cc:1468] Processing CreateTablet for tablet c56b761dc3b74621ac786ac1d786d539 (DEFAULT_TABLE table=default.Table_c4f60a9a65f54ed781a9b1643ba15c34 [id=397a102cc27743f18ddf17277d5b10b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.368518 890 tablet_service.cc:1468] Processing CreateTablet for tablet 969cffe4ef854409a3853a1c1d0d9a9a (DEFAULT_TABLE table=default.Table_8ea631c388eb4b60923f367cd108d236 [id=15079533f87d460f97e5ae86792b35eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.369414 896 tablet_service.cc:1468] Processing CreateTablet for tablet 1a8dddc0659a4340af7f2d01c9a3151f (DEFAULT_TABLE table=default.Table_aa4b088ab043449280d641b0ff03d845 [id=aa724b93d430458095befa5f20b1ef70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.370554 879 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba02adec0d74547ab12dff3a0c898f7 (DEFAULT_TABLE table=default.Table_36921981749241789ba5b79d109b0549 [id=48e4102eef824437b95a71f60c5c1ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.370671 2217 raft_consensus.cc:491] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.378412 2217 raft_consensus.cc:513] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.378727 1157 ts_tablet_manager.cc:1905] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.378880 2217 leader_election.cc:290] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.371524 889 tablet_service.cc:1468] Processing CreateTablet for tablet 13d779335d09442682554322b59f915a (DEFAULT_TABLE table=default.Table_2698ae66abbc4163beb2f3939554be2a [id=bce6f6d30f594035b37966090fcf9d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.375236 883 tablet_service.cc:1468] Processing CreateTablet for tablet ea12171996e6408da1af35e325233e7d (DEFAULT_TABLE table=default.Table_bd359023e99a4eecbd2606dd52146994 [id=6b38eba78f534170a0401b08b310f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.379850 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
W20250629 01:51:55.380340 850 leader_election.cc:343] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 75ce1bd4af434f7e99092aafc0ab0118
I20250629 01:51:55.380437 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ecf731c6cfa4656acc992e2f8e06c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.382591 2273 raft_consensus.cc:2802] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:55.382723 2273 raft_consensus.cc:695] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:55.382943 2273 consensus_queue.cc:237] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:55.385447 850 leader_election.cc:343] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 75ce1bd4af434f7e99092aafc0ab0118
I20250629 01:51:55.385540 850 leader_election.cc:304] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.386377 2217 raft_consensus.cc:2747] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.387868 1030 tablet_service.cc:1468] Processing CreateTablet for tablet f1ad1b7de79b48058437a9dea5d30b37 (DEFAULT_TABLE table=default.Table_97c5093f3a1542f5bb7fc463c1955f9e [id=6faf9bc8cce84174a48928674dc25702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.388079 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 7cefc4de48e844f496459d7ae966ab52 (DEFAULT_TABLE table=default.Table_0814dc6691b040e1a9f69f258973cf28 [id=896fee5246524ebabd2f166ae669fefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.388255 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1ad1b7de79b48058437a9dea5d30b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.388499 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cefc4de48e844f496459d7ae966ab52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.388674 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c4530a43bdeb47d3ad29b17750be12d2 (DEFAULT_TABLE table=default.Table_7cf9a31f57704fea83af6665be37020e [id=07f8379f2af6486995817a514e02130a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.388954 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4530a43bdeb47d3ad29b17750be12d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.390130 1159 tablet_bootstrap.cc:492] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.390520 1159 ts_tablet_manager.cc:1397] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.000s
I20250629 01:51:55.392309 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56b761dc3b74621ac786ac1d786d539. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.393469 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c1b5deb7f2204e3aa271ddd818a360d3 (DEFAULT_TABLE table=default.Table_5d60769b6bad4b8eb8dfa90c3047c054 [id=777823b736a74d3585be9f11f0ffe055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.393785 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1b5deb7f2204e3aa271ddd818a360d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.396255 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969cffe4ef854409a3853a1c1d0d9a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.396417 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 72f277d4234744a2995953a5eb1715b3 (DEFAULT_TABLE table=default.Table_b5a50c642fa1422bb612d0edeadb4ee7 [id=e3d515b8692445e5afdea63aeadedc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.396668 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f277d4234744a2995953a5eb1715b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.396704 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.397073 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 1ecf731c6cfa4656acc992e2f8e06c08 (DEFAULT_TABLE table=default.Table_6a6e2363eabb4ce18ec597b680b149d5 [id=94c53c71b4c140dbaa88bb9984a26e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.397318 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ecf731c6cfa4656acc992e2f8e06c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.396464 1159 raft_consensus.cc:357] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.397907 1159 raft_consensus.cc:383] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.398484 1159 raft_consensus.cc:738] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.399706 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 30265f664bff4b058e587edd9e0c5978 (DEFAULT_TABLE table=default.Table_9a012ad660f649da977b3123b27ecd2e [id=43c1739e8c674274a89bbfc16f16e3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.399981 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30265f664bff4b058e587edd9e0c5978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.396397 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9479685385b5402b9ec21249621b1600 (DEFAULT_TABLE table=default.Table_eee311f8e86f4b1da839f4f16aa2b343 [id=ff22b7d417a741559c91ef3360e0d50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.401801 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9479685385b5402b9ec21249621b1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.402029 1015 tablet_service.cc:1468] Processing CreateTablet for tablet c56b761dc3b74621ac786ac1d786d539 (DEFAULT_TABLE table=default.Table_c4f60a9a65f54ed781a9b1643ba15c34 [id=397a102cc27743f18ddf17277d5b10b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.402253 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56b761dc3b74621ac786ac1d786d539. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.401130 1159 consensus_queue.cc:260] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.403064 1158 tablet_replica.cc:331] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:55.403225 1158 raft_consensus.cc:2241] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.403342 1158 raft_consensus.cc:2270] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.403800 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a8dddc0659a4340af7f2d01c9a3151f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.403951 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13d779335d09442682554322b59f915a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.404235 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea12171996e6408da1af35e325233e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.406725 1159 ts_tablet_manager.cc:1428] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250629 01:51:55.408259 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba02adec0d74547ab12dff3a0c898f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.409384 1159 tablet_bootstrap.cc:492] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.410187 1158 ts_tablet_manager.cc:1905] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.412578 1155 tablet_bootstrap.cc:492] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.412674 1155 ts_tablet_manager.cc:1397] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.005s
I20250629 01:51:55.413239 1155 raft_consensus.cc:357] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.413357 1155 raft_consensus.cc:383] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.413395 1155 raft_consensus.cc:738] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.413545 1155 consensus_queue.cc:260] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.414027 1155 ts_tablet_manager.cc:1428] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:55.414237 1155 tablet_bootstrap.cc:492] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.416949 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.417093 1052 raft_consensus.cc:2466] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:55.418376 2217 raft_consensus.cc:491] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.418493 2217 raft_consensus.cc:513] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.418601 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.418758 1052 raft_consensus.cc:2466] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:55.418939 2217 leader_election.cc:290] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:55.419387 2271 tablet.cc:1722] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:55.419620 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:55.419735 1052 raft_consensus.cc:3058] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.421831 1052 raft_consensus.cc:2466] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:55.423012 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.422998 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a6bd5d27120c4ca28bdd2f5ebc2170d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.423602 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6bd5d27120c4ca28bdd2f5ebc2170d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.424453 1158 ts_tablet_manager.cc:1918] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.424576 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 4ba02adec0d74547ab12dff3a0c898f7 (DEFAULT_TABLE table=default.Table_36921981749241789ba5b79d109b0549 [id=48e4102eef824437b95a71f60c5c1ca7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.424713 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "943f584fb9554fa4b65e5302443cec92" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:55.424866 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba02adec0d74547ab12dff3a0c898f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.425251 1159 tablet_bootstrap.cc:654] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.419588 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 13d779335d09442682554322b59f915a (DEFAULT_TABLE table=default.Table_2698ae66abbc4163beb2f3939554be2a [id=bce6f6d30f594035b37966090fcf9d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.426018 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13d779335d09442682554322b59f915a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.426965 850 leader_election.cc:304] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.427500 2217 raft_consensus.cc:2747] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.433044 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 1a8dddc0659a4340af7f2d01c9a3151f (DEFAULT_TABLE table=default.Table_aa4b088ab043449280d641b0ff03d845 [id=aa724b93d430458095befa5f20b1ef70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.433313 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a8dddc0659a4340af7f2d01c9a3151f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.436062 1154 tablet_bootstrap.cc:492] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.436226 1154 ts_tablet_manager.cc:1397] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.139s user 0.001s sys 0.011s
I20250629 01:51:55.436837 2049 raft_consensus.cc:491] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.436846 1154 raft_consensus.cc:357] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.436957 1154 raft_consensus.cc:383] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.436993 1154 raft_consensus.cc:738] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.436954 2049 raft_consensus.cc:513] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.437151 1154 consensus_queue.cc:260] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.437439 2049 leader_election.cc:290] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.437484 1154 ts_tablet_manager.cc:1428] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:55.437669 1154 tablet_bootstrap.cc:492] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.439343 1154 tablet_bootstrap.cc:654] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.439342 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da93c730451d42bc8df1e582fb3af818" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.439533 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ba02adec0d74547ab12dff3a0c898f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.439800 712 leader_election.cc:343] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.439350 1157 ts_tablet_manager.cc:1918] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.440029 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13d779335d09442682554322b59f915a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.442655 1155 tablet_bootstrap.cc:654] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.444878 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a8dddc0659a4340af7f2d01c9a3151f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.450704 916 raft_consensus.cc:3058] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.440044 1157 log.cc:1199] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/942f7bbe29f94083b2494aab8fbcc0c5
I20250629 01:51:55.452378 1157 ts_tablet_manager.cc:1939] T 942f7bbe29f94083b2494aab8fbcc0c5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.452832 1157 tablet_replica.cc:331] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.453080 1157 raft_consensus.cc:2241] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.454278 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 942f7bbe29f94083b2494aab8fbcc0c5 (table default.Table_d5a40734fce446ae9ad039d5928a7977 [id=05a633be0bb74a1388daba48dd2bfb7e]) successfully deleted
I20250629 01:51:55.454366 1157 raft_consensus.cc:2270] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.454941 1157 ts_tablet_manager.cc:1905] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.456516 754 tablet_service.cc:1468] Processing CreateTablet for tablet ea12171996e6408da1af35e325233e7d (DEFAULT_TABLE table=default.Table_bd359023e99a4eecbd2606dd52146994 [id=6b38eba78f534170a0401b08b310f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.456785 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea12171996e6408da1af35e325233e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.459535 749 tablet_service.cc:1468] Processing CreateTablet for tablet ab8cf887e2404bd38029df6aebaa363d (DEFAULT_TABLE table=default.Table_5cd8e243b623461eabaa41b2b6d9988a [id=8d6ef0a09ae34bb298bc7c221a91b0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.459787 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8cf887e2404bd38029df6aebaa363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.464143 1159 tablet_bootstrap.cc:492] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.464336 1159 ts_tablet_manager.cc:1397] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20250629 01:51:55.465201 1159 raft_consensus.cc:357] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.465462 1159 raft_consensus.cc:383] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.465601 1159 raft_consensus.cc:738] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.465947 1159 consensus_queue.cc:260] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.466787 1159 ts_tablet_manager.cc:1428] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:55.466933 743 tablet_service.cc:1468] Processing CreateTablet for tablet edbc80f0be2e409cb2b9f2bf206c944a (DEFAULT_TABLE table=default.Table_6d323c26638e4b2c9a01abd8feee7412 [id=10faf1f9a83a4b59bb0fc65c586f4a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.467278 1159 tablet_bootstrap.cc:492] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.467278 746 tablet_service.cc:1468] Processing CreateTablet for tablet 790a1347e57f4cea895132564eafae9c (DEFAULT_TABLE table=default.Table_27f4e21d29ed426089885ef45a154384 [id=a531d97bb9a046dab631cef3c2451a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.467288 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edbc80f0be2e409cb2b9f2bf206c944a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.467721 739 tablet_service.cc:1468] Processing CreateTablet for tablet 2e9e6727c5e3422ab1463298cf3ec974 (DEFAULT_TABLE table=default.Table_583444e245fc499bb4f38e7921e59da3 [id=361bf326f3cb4769be08970f7900d7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.467989 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9e6727c5e3422ab1463298cf3ec974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.468330 742 tablet_service.cc:1468] Processing CreateTablet for tablet 0fe420d8b8bd4817a10677b8a5eadd32 (DEFAULT_TABLE table=default.Table_3581bbb91ff84108a35412b25174b190 [id=86281322c3fb478597dd48de8f48f815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.468587 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe420d8b8bd4817a10677b8a5eadd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.469961 754 tablet_service.cc:1468] Processing CreateTablet for tablet fa2052970beb4e769b61d7cd3edc0db6 (DEFAULT_TABLE table=default.Table_cc8b682489c3440583a2b90e20fe6e79 [id=34891731158e4cbfbd459043105edfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.470341 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2052970beb4e769b61d7cd3edc0db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.472307 741 tablet_service.cc:1468] Processing CreateTablet for tablet 64a4a34fadc44bf582fab2419a067fa5 (DEFAULT_TABLE table=default.Table_87ba700594ec433ebf0252c79cdbdab9 [id=ed5bc9dcb04a4701b2c368255c98ab07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.472575 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a4a34fadc44bf582fab2419a067fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.476274 757 tablet_service.cc:1468] Processing CreateTablet for tablet 22e4688882424793a25b9627466e1214 (DEFAULT_TABLE table=default.Table_5829573f0cc6418ab66487ff0902cab5 [id=61822f27d597481fa72ddc2b5805f8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.476517 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22e4688882424793a25b9627466e1214. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.476462 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ee1baa976b5e47e490a0fd561b37f83d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1baa976b5e47e490a0fd561b37f83d already in progress: creating tablet
W20250629 01:51:55.476792 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet caa63fe10ff9487a9fc16479285c5abd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caa63fe10ff9487a9fc16479285c5abd already in progress: creating tablet
W20250629 01:51:55.476985 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8cea647451954d8ab3bd495a3a080c97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cea647451954d8ab3bd495a3a080c97 already in progress: creating tablet
W20250629 01:51:55.477150 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 06d591c66d734f76a1bda2194ec62776 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06d591c66d734f76a1bda2194ec62776 already in progress: creating tablet
I20250629 01:51:55.480316 1159 tablet_bootstrap.cc:654] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:55.480314 710 leader_election.cc:343] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9bbbeb43473a4f6f892056a30f62589f
I20250629 01:51:55.480401 710 leader_election.cc:304] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:55.480715 710 leader_election.cc:304] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.480908 2228 raft_consensus.cc:2747] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.481063 2228 raft_consensus.cc:2802] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:55.481096 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790a1347e57f4cea895132564eafae9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.481114 2228 raft_consensus.cc:491] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:55.481163 2228 raft_consensus.cc:3058] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.483243 2228 raft_consensus.cc:513] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.483739 2228 leader_election.cc:290] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.484304 710 leader_election.cc:304] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.484817 2052 consensus_queue.cc:1035] T d159e69137d34899ace5c245638a3c0f P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:55.485177 2052 raft_consensus.cc:2802] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:55.485230 2052 raft_consensus.cc:491] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:55.485273 2052 raft_consensus.cc:3058] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.487272 2052 raft_consensus.cc:513] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.491356 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:55.491804 712 leader_election.cc:343] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.492079 2052 leader_election.cc:290] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.492527 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:55.492933 712 leader_election.cc:343] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.493077 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ea12171996e6408da1af35e325233e7d (DEFAULT_TABLE table=default.Table_bd359023e99a4eecbd2606dd52146994 [id=6b38eba78f534170a0401b08b310f122]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.493402 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea12171996e6408da1af35e325233e7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.493832 879 tablet_service.cc:1468] Processing CreateTablet for tablet edbc80f0be2e409cb2b9f2bf206c944a (DEFAULT_TABLE table=default.Table_6d323c26638e4b2c9a01abd8feee7412 [id=10faf1f9a83a4b59bb0fc65c586f4a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.494000 889 tablet_service.cc:1468] Processing CreateTablet for tablet 2e9e6727c5e3422ab1463298cf3ec974 (DEFAULT_TABLE table=default.Table_583444e245fc499bb4f38e7921e59da3 [id=361bf326f3cb4769be08970f7900d7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.494218 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edbc80f0be2e409cb2b9f2bf206c944a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.494397 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9e6727c5e3422ab1463298cf3ec974. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.501108 1155 tablet_bootstrap.cc:492] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.501834 1155 ts_tablet_manager.cc:1397] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.002s
I20250629 01:51:55.503684 883 tablet_service.cc:1468] Processing CreateTablet for tablet 22e4688882424793a25b9627466e1214 (DEFAULT_TABLE table=default.Table_5829573f0cc6418ab66487ff0902cab5 [id=61822f27d597481fa72ddc2b5805f8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.503727 895 tablet_service.cc:1468] Processing CreateTablet for tablet 64a4a34fadc44bf582fab2419a067fa5 (DEFAULT_TABLE table=default.Table_87ba700594ec433ebf0252c79cdbdab9 [id=ed5bc9dcb04a4701b2c368255c98ab07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.505982 881 tablet_service.cc:1468] Processing CreateTablet for tablet ab8cf887e2404bd38029df6aebaa363d (DEFAULT_TABLE table=default.Table_5cd8e243b623461eabaa41b2b6d9988a [id=8d6ef0a09ae34bb298bc7c221a91b0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.506328 890 tablet_service.cc:1468] Processing CreateTablet for tablet 790a1347e57f4cea895132564eafae9c (DEFAULT_TABLE table=default.Table_27f4e21d29ed426089885ef45a154384 [id=a531d97bb9a046dab631cef3c2451a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.506328 2238 tablet.cc:1722] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:55.506700 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 969cffe4ef854409a3853a1c1d0d9a9a (DEFAULT_TABLE table=default.Table_8ea631c388eb4b60923f367cd108d236 [id=15079533f87d460f97e5ae86792b35eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.506980 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969cffe4ef854409a3853a1c1d0d9a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.507550 2283 tablet.cc:1722] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:55.507615 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ab8cf887e2404bd38029df6aebaa363d (DEFAULT_TABLE table=default.Table_5cd8e243b623461eabaa41b2b6d9988a [id=8d6ef0a09ae34bb298bc7c221a91b0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.507879 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8cf887e2404bd38029df6aebaa363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.508553 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a4a34fadc44bf582fab2419a067fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.508741 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22e4688882424793a25b9627466e1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.510892 888 tablet_service.cc:1468] Processing CreateTablet for tablet fa2052970beb4e769b61d7cd3edc0db6 (DEFAULT_TABLE table=default.Table_cc8b682489c3440583a2b90e20fe6e79 [id=34891731158e4cbfbd459043105edfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.511461 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790a1347e57f4cea895132564eafae9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.511833 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8cf887e2404bd38029df6aebaa363d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.512601 1019 tablet_service.cc:1468] Processing CreateTablet for tablet fa2052970beb4e769b61d7cd3edc0db6 (DEFAULT_TABLE table=default.Table_cc8b682489c3440583a2b90e20fe6e79 [id=34891731158e4cbfbd459043105edfef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.512898 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2052970beb4e769b61d7cd3edc0db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.514060 2217 raft_consensus.cc:491] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.514185 2217 raft_consensus.cc:513] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.514698 2217 leader_election.cc:290] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:55.515290 850 leader_election.cc:343] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e962a020664b42ea918e5952dc43aade
I20250629 01:51:55.516265 889 tablet_service.cc:1468] Processing CreateTablet for tablet 0fe420d8b8bd4817a10677b8a5eadd32 (DEFAULT_TABLE table=default.Table_3581bbb91ff84108a35412b25174b190 [id=86281322c3fb478597dd48de8f48f815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.516575 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe420d8b8bd4817a10677b8a5eadd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.518172 1019 tablet_service.cc:1468] Processing CreateTablet for tablet edbc80f0be2e409cb2b9f2bf206c944a (DEFAULT_TABLE table=default.Table_6d323c26638e4b2c9a01abd8feee7412 [id=10faf1f9a83a4b59bb0fc65c586f4a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.518437 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edbc80f0be2e409cb2b9f2bf206c944a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.520114 1158 log.cc:1199] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:51:55.520417 1158 ts_tablet_manager.cc:1939] T 8a880c9bb4b94d3b9b12f28d48565f06 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:55.524353 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 0fe420d8b8bd4817a10677b8a5eadd32 (DEFAULT_TABLE table=default.Table_3581bbb91ff84108a35412b25174b190 [id=86281322c3fb478597dd48de8f48f815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.524641 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe420d8b8bd4817a10677b8a5eadd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.525116 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 2e9e6727c5e3422ab1463298cf3ec974 (DEFAULT_TABLE table=default.Table_583444e245fc499bb4f38e7921e59da3 [id=361bf326f3cb4769be08970f7900d7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.525345 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9e6727c5e3422ab1463298cf3ec974. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.527319 850 leader_election.cc:343] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e962a020664b42ea918e5952dc43aade
I20250629 01:51:55.527406 850 leader_election.cc:304] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.527671 740 tablet_service.cc:1468] Processing CreateTablet for tablet 4c07a1580c734aeba695151d23c7e020 (DEFAULT_TABLE table=default.Table_a205f56d1700435b9739b3651efae052 [id=8ce0fe9e8b0f46fcbcaa622b9652c1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.527837 2217 raft_consensus.cc:2747] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.527959 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c07a1580c734aeba695151d23c7e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.528566 883 tablet_service.cc:1468] Processing CreateTablet for tablet 4c07a1580c734aeba695151d23c7e020 (DEFAULT_TABLE table=default.Table_a205f56d1700435b9739b3651efae052 [id=8ce0fe9e8b0f46fcbcaa622b9652c1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.529515 2217 raft_consensus.cc:491] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.529628 2217 raft_consensus.cc:513] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.529978 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 64a4a34fadc44bf582fab2419a067fa5 (DEFAULT_TABLE table=default.Table_87ba700594ec433ebf0252c79cdbdab9 [id=ed5bc9dcb04a4701b2c368255c98ab07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.530078 2217 leader_election.cc:290] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.530262 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a4a34fadc44bf582fab2419a067fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.530718 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:55.530782 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b6fa908ded8240bb962674e1efb29c74" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.531368 850 leader_election.cc:304] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.531647 2217 raft_consensus.cc:2747] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.532923 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2660c211d65f41b0a50b5a6f7d8be48a (DEFAULT_TABLE table=default.Table_d5002228f791426d81f36cbba806a047 [id=6765021c345c49709ee6edfa8376142b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.533581 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2660c211d65f41b0a50b5a6f7d8be48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.532806 1155 raft_consensus.cc:357] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.535866 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c07a1580c734aeba695151d23c7e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.536054 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa2052970beb4e769b61d7cd3edc0db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.536176 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 22e4688882424793a25b9627466e1214 (DEFAULT_TABLE table=default.Table_5829573f0cc6418ab66487ff0902cab5 [id=61822f27d597481fa72ddc2b5805f8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.536463 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22e4688882424793a25b9627466e1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.367547 745 tablet_service.cc:1468] Processing CreateTablet for tablet 1ecf731c6cfa4656acc992e2f8e06c08 (DEFAULT_TABLE table=default.Table_6a6e2363eabb4ce18ec597b680b149d5 [id=94c53c71b4c140dbaa88bb9984a26e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.540849 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 4c07a1580c734aeba695151d23c7e020 (DEFAULT_TABLE table=default.Table_a205f56d1700435b9739b3651efae052 [id=8ce0fe9e8b0f46fcbcaa622b9652c1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.541133 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c07a1580c734aeba695151d23c7e020. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.535079 1155 raft_consensus.cc:383] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.541857 1155 raft_consensus.cc:738] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
W20250629 01:51:55.546404 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 61829.1us
I20250629 01:51:55.546566 710 leader_election.cc:304] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.546753 2049 raft_consensus.cc:2802] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:55.546838 2049 raft_consensus.cc:695] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:55.541007 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ecf731c6cfa4656acc992e2f8e06c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.547011 2049 consensus_queue.cc:237] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:55.549777 710 leader_election.cc:343] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 1c995ce14c9a4966b842fcba520cf87d
I20250629 01:51:55.549854 710 leader_election.cc:304] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:55.550496 2052 raft_consensus.cc:2747] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.553736 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9ed1ee73bb8f4761806a66f988b57bf9
I20250629 01:51:55.554288 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.554880 881 tablet_service.cc:1468] Processing CreateTablet for tablet 2660c211d65f41b0a50b5a6f7d8be48a (DEFAULT_TABLE table=default.Table_d5002228f791426d81f36cbba806a047 [id=6765021c345c49709ee6edfa8376142b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.555166 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2660c211d65f41b0a50b5a6f7d8be48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.556649 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 303f77a729c648f9b1146d11056b242d
I20250629 01:51:55.557194 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 303f77a729c648f9b1146d11056b242d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.557449 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3ba5887dffc1460a89208447ecc6a198
I20250629 01:51:55.544579 1155 consensus_queue.cc:260] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.558000 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba5887dffc1460a89208447ecc6a198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:55.558218 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22e4688882424793a25b9627466e1214 consensus state has no leader
I20250629 01:51:55.558621 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2521a476643c41c1a2e5282755a58091
I20250629 01:51:55.559257 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 2521a476643c41c1a2e5282755a58091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.559291 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a2e9de48e4ad401880f351f2bd55cf5c
I20250629 01:51:55.561230 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a2e9de48e4ad401880f351f2bd55cf5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.564297 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e5de441f126149e9810a9725612afbd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.565025 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3bd1be2b90da48c7aed1e37d5508c9c1" } new_table_name: "default.Table_9823577beb5242a798718da814e438bf" modify_external_catalogs: true
I20250629 01:51:55.565253 1159 tablet_bootstrap.cc:492] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.564925 1155 ts_tablet_manager.cc:1428] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20250629 01:51:55.565778 2238 tablet.cc:1722] T 943f584fb9554fa4b65e5302443cec92 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:55.567265 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_78bb87e44b9e4ceea207030a95646888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.568253 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce] alter complete (version 3)
W20250629 01:51:55.571560 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a876b3db3904a4297b5b13a9325b1c3 consensus state has no leader
I20250629 01:51:55.573817 455 catalog_manager.cc:5582] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:55.574199 455 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:55.574350 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:55.575699 2283 tablet.cc:1722] T 943f584fb9554fa4b65e5302443cec92 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:51:55.575862 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5de441f126149e9810a9725612afbd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:55.577356 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78bb87e44b9e4ceea207030a95646888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.577520 1159 ts_tablet_manager.cc:1397] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.110s user 0.000s sys 0.006s
I20250629 01:51:55.578792 1155 tablet_bootstrap.cc:492] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.580415 2217 raft_consensus.cc:491] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.580513 2217 raft_consensus.cc:513] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.580967 2217 leader_election.cc:290] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:55.585698 1159 raft_consensus.cc:357] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.591076 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 2521a476643c41c1a2e5282755a58091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:55.591171 850 leader_election.cc:343] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 47d4287317ec4788bd96ad174c31748d
I20250629 01:51:55.591013 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba5887dffc1460a89208447ecc6a198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:55.591775 850 leader_election.cc:343] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 47d4287317ec4788bd96ad174c31748d
I20250629 01:51:55.592151 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a2e9de48e4ad401880f351f2bd55cf5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.591894 850 leader_election.cc:304] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.592502 1159 raft_consensus.cc:383] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.592527 2217 raft_consensus.cc:2747] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.593915 1159 raft_consensus.cc:738] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.594476 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]). Expected version 1 got 0
I20250629 01:51:55.593047 1157 ts_tablet_manager.cc:1918] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.594722 1159 consensus_queue.cc:260] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.596217 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 303f77a729c648f9b1146d11056b242d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.596393 888 tablet_service.cc:1515] Processing DeleteTablet for tablet e83009c1599a43d18d1c73c0101ab0be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.597275 1154 tablet_bootstrap.cc:492] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.597445 1154 ts_tablet_manager.cc:1397] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.160s user 0.011s sys 0.048s
I20250629 01:51:55.598081 1154 raft_consensus.cc:357] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.598150 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.598248 1154 raft_consensus.cc:383] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.598312 1154 raft_consensus.cc:738] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.598489 1154 consensus_queue.cc:260] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.596032 1157 log.cc:1199] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/35990d206ba64159ab05ab7b7b19d064
I20250629 01:51:55.600641 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_4226edee39d24dafb4dac54e8c6c7c58 [id=4bfe4f0aff75478fbd422f8c1f46d45e] alter complete (version 1)
I20250629 01:51:55.602295 1154 ts_tablet_manager.cc:1428] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:55.604455 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 2660c211d65f41b0a50b5a6f7d8be48a (DEFAULT_TABLE table=default.Table_d5002228f791426d81f36cbba806a047 [id=6765021c345c49709ee6edfa8376142b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.604738 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2660c211d65f41b0a50b5a6f7d8be48a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.604959 712 consensus_peers.cc:489] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:55.608814 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 303f77a729c648f9b1146d11056b242d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.608942 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba5887dffc1460a89208447ecc6a198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.609001 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 2521a476643c41c1a2e5282755a58091 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.609098 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a2e9de48e4ad401880f351f2bd55cf5c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.609793 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.611948 1156 ts_tablet_manager.cc:1918] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:55.612478 710 leader_election.cc:343] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: da93c730451d42bc8df1e582fb3af818
I20250629 01:51:55.612614 710 leader_election.cc:304] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:55.612830 2052 raft_consensus.cc:2747] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.613158 1154 tablet_bootstrap.cc:492] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.614235 1154 tablet_bootstrap.cc:654] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.614899 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_b4abe11363964e61a884d8bf8f8d0189 [id=15f7a887c8224335aaf55ad0469bbf12] alter complete (version 1)
I20250629 01:51:55.615665 1155 tablet_bootstrap.cc:654] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.615926 1159 ts_tablet_manager.cc:1428] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20250629 01:51:55.615480 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:55.617456 1157 ts_tablet_manager.cc:1939] T 35990d206ba64159ab05ab7b7b19d064 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.619679 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8a880c9bb4b94d3b9b12f28d48565f06 (table default.Table_0ed9b0a279de49078207546c6ddbc21f [id=4457fae28052479e81efac2f717c1bb6]) successfully deleted
I20250629 01:51:55.621431 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 35990d206ba64159ab05ab7b7b19d064 (table default.Table_c6e85ae2e824442fa3e307eda6dc1aa2 [id=8d49193b68504e77818e373ecad346b6]) successfully deleted
I20250629 01:51:55.625288 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.625471 1159 tablet_bootstrap.cc:492] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:55.626595 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5d2f2975e5d64efd888ed948f1420916 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2f2975e5d64efd888ed948f1420916 already in progress: creating tablet
I20250629 01:51:55.628751 1157 tablet_replica.cc:331] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.629565 1157 raft_consensus.cc:2241] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.630087 1157 raft_consensus.cc:2270] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.630338 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 790a1347e57f4cea895132564eafae9c (DEFAULT_TABLE table=default.Table_27f4e21d29ed426089885ef45a154384 [id=a531d97bb9a046dab631cef3c2451a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.630640 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 790a1347e57f4cea895132564eafae9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.612071 1156 log.cc:1199] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a40c1a8eec9549a8bdf695d6516440a8
I20250629 01:51:55.632381 1156 ts_tablet_manager.cc:1939] T a40c1a8eec9549a8bdf695d6516440a8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:55.632459 1157 ts_tablet_manager.cc:1905] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.633049 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 998) took 1448 ms. Trace:
I20250629 01:51:55.633149 1156 rpcz_store.cc:276] 0629 01:51:54.184657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.184710 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:55.360851 (+1176141us) ts_tablet_manager.cc:1145] Deleting tablet a40c1a8eec9549a8bdf695d6516440a8
0629 01:51:55.360859 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.553364 (+192505us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.611873 (+ 58509us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.633032 (+ 21159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175924}
I20250629 01:51:55.633531 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a40c1a8eec9549a8bdf695d6516440a8 (table default.Table_6f6d8cf186fc4934ac5690bac0754373 [id=db3232e7e81240a699e87310dc0627d3]) successfully deleted
I20250629 01:51:55.633426 1159 tablet_bootstrap.cc:654] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.637173 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:55.637352 1049 raft_consensus.cc:3058] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.638350 1156 tablet_replica.cc:331] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:55.639869 1156 raft_consensus.cc:2241] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.640036 1156 raft_consensus.cc:2270] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.640491 1156 ts_tablet_manager.cc:1905] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.640594 1049 raft_consensus.cc:2466] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:55.641660 1154 tablet_bootstrap.cc:492] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.641817 1154 ts_tablet_manager.cc:1397] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20250629 01:51:55.642462 1154 raft_consensus.cc:357] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.642630 1154 raft_consensus.cc:383] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.642707 1154 raft_consensus.cc:738] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.643090 1154 consensus_queue.cc:260] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.643908 2271 tablet.cc:1722] T 943f584fb9554fa4b65e5302443cec92 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:55.645169 1154 ts_tablet_manager.cc:1428] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:55.645381 1154 tablet_bootstrap.cc:492] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.647360 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:55.647511 1052 raft_consensus.cc:3058] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.648057 1154 tablet_bootstrap.cc:654] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.647542 1159 tablet_bootstrap.cc:492] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.649713 1052 raft_consensus.cc:2466] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:55.649883 1159 ts_tablet_manager.cc:1397] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250629 01:51:55.653740 1155 tablet_bootstrap.cc:492] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.653904 1155 ts_tablet_manager.cc:1397] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.002s
I20250629 01:51:55.654601 1155 raft_consensus.cc:357] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.654836 1155 raft_consensus.cc:383] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.654937 1155 raft_consensus.cc:738] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.655155 1155 consensus_queue.cc:260] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.655642 1155 ts_tablet_manager.cc:1428] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:55.658041 1156 ts_tablet_manager.cc:1918] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.660308 1156 log.cc:1199] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b6d21dfd48ea4a95a2fcecde6735974b
I20250629 01:51:55.660573 1156 ts_tablet_manager.cc:1939] T b6d21dfd48ea4a95a2fcecde6735974b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:55.661159 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 996) took 1476 ms. Trace:
I20250629 01:51:55.661253 1156 rpcz_store.cc:276] 0629 01:51:54.184462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.184880 (+ 418us) service_pool.cc:225] Handling call
0629 01:51:55.638313 (+1453433us) ts_tablet_manager.cc:1145] Deleting tablet b6d21dfd48ea4a95a2fcecde6735974b
0629 01:51:55.638320 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.650518 (+ 12198us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.657976 (+ 7458us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.661144 (+ 3168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1453292}
I20250629 01:51:55.661639 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b6d21dfd48ea4a95a2fcecde6735974b (table default.Table_1b37a535cd2f483a8c72aa10882a4301 [id=b516da4148384818bcc69e4259f20ef9]) successfully deleted
I20250629 01:51:55.661897 1156 tablet_replica.cc:331] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:55.662070 1156 raft_consensus.cc:2241] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.662699 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_60c92ddc18dd4bab980e43d13e027a00" } new_table_name: "default.Table_3592b5e7e0c44766a622a04a3a5c9b03" modify_external_catalogs: true
I20250629 01:51:55.663952 1159 raft_consensus.cc:357] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.664960 1159 raft_consensus.cc:383] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.665306 1159 raft_consensus.cc:738] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.665953 1156 raft_consensus.cc:2270] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.663102 1155 tablet_bootstrap.cc:492] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.667256 1155 tablet_bootstrap.cc:654] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.666930 1159 consensus_queue.cc:260] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.668267 1156 ts_tablet_manager.cc:1905] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:55.669071 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82a837932f14419c8b32fcd2c9358215 consensus state has no leader
I20250629 01:51:55.671237 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6d58bedddad3413d9c67e6aa0caaa6cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.671617 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2f55e3c1a16e4082aaacb9434d6d583c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.671823 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2fa9db42a28d4da8bb5c062c63e8098b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.672322 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.674106 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3911d6bceecf451cb29637a4843b711a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:55.674667 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 241a5c9aee3f4297b67b18380ef5ef88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.674649 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1048b0cf35964b1fb36e7004e009ac0e" } new_table_name: "default.Table_684371eb22934033bf7b4904eec13f8d" modify_external_catalogs: true
I20250629 01:51:55.675669 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bd359023e99a4eecbd2606dd52146994" } new_table_name: "default.Table_75fe4e5ef4544f3b99c5567decbce1c6" modify_external_catalogs: true
I20250629 01:51:55.677271 1156 ts_tablet_manager.cc:1918] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.678406 1159 ts_tablet_manager.cc:1428] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
W20250629 01:51:55.683341 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f55e3c1a16e4082aaacb9434d6d583c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:55.683452 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fa9db42a28d4da8bb5c062c63e8098b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:55.684737 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d58bedddad3413d9c67e6aa0caaa6cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.685163 1156 log.cc:1199] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/56b9ef027c9d42048e4a6dff221df793
I20250629 01:51:55.685415 1156 ts_tablet_manager.cc:1939] T 56b9ef027c9d42048e4a6dff221df793 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:55.685597 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.685631 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 995) took 1501 ms. Trace:
I20250629 01:51:55.685691 1156 rpcz_store.cc:276] 0629 01:51:54.183823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.185230 (+ 1407us) service_pool.cc:225] Handling call
0629 01:51:55.661860 (+1476630us) ts_tablet_manager.cc:1145] Deleting tablet 56b9ef027c9d42048e4a6dff221df793
0629 01:51:55.661868 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.674829 (+ 12961us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.677211 (+ 2382us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.685619 (+ 8408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476488}
I20250629 01:51:55.686043 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 56b9ef027c9d42048e4a6dff221df793 (table default.Table_5149d5ed90ca4d6ca10f30e1694a0e1d [id=24d959060a484a2bab405fa65b92abf0]) successfully deleted
I20250629 01:51:55.686105 1159 tablet_bootstrap.cc:492] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:55.686407 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3911d6bceecf451cb29637a4843b711a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:55.687386 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea12171996e6408da1af35e325233e7d consensus state has no leader
W20250629 01:51:55.687919 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df0fe7f5469d44fc8f6ac5c72377b48a consensus state has no leader
I20250629 01:51:55.688853 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 994) took 1505 ms. Trace:
I20250629 01:51:55.689121 1156 rpcz_store.cc:276] 0629 01:51:54.183499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.193481 (+ 9982us) service_pool.cc:225] Handling call
0629 01:51:55.688551 (+1495070us) ts_tablet_manager.cc:1145] Deleting tablet 31cee0e741584157b0333851f94c8559
0629 01:51:55.688558 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.688834 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494566}
I20250629 01:51:55.689666 1156 tablet_replica.cc:331] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:55.689960 1156 raft_consensus.cc:2241] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.690271 1156 raft_consensus.cc:2270] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.690368 2273 raft_consensus.cc:491] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.690490 2273 raft_consensus.cc:513] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.690788 1156 ts_tablet_manager.cc:1905] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.691040 2273 leader_election.cc:290] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:55.691042 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 31cee0e741584157b0333851f94c8559 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31cee0e741584157b0333851f94c8559 already in progress: creating tablet
I20250629 01:51:55.691766 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47cbc958eca14ed69fb45c3c2984aeef" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.692031 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47cbc958eca14ed69fb45c3c2984aeef" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:55.692158 988 leader_election.cc:343] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:55.692440 986 leader_election.cc:343] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.692531 986 leader_election.cc:304] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.692827 2273 raft_consensus.cc:2747] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.693885 2298 raft_consensus.cc:491] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.694022 2298 raft_consensus.cc:513] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.694700 2049 raft_consensus.cc:491] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.694736 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52be184d0480495a894442d6f72a0f12" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.694823 2049 raft_consensus.cc:513] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:55.695145 712 leader_election.cc:343] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.695377 2298 leader_election.cc:290] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.695652 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9f59fbe9634ac8b27f990d9496f637" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.696121 712 leader_election.cc:343] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.696270 2049 leader_election.cc:290] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.701834 1159 tablet_bootstrap.cc:654] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.699784 1157 ts_tablet_manager.cc:1918] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.702513 1157 log.cc:1199] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e962a020664b42ea918e5952dc43aade
I20250629 01:51:55.705164 2217 raft_consensus.cc:491] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.705528 2217 raft_consensus.cc:513] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.706120 2217 leader_election.cc:290] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.706418 451 master_service.cc:948] ReplaceTablet: received request to replace tablet f5941dd6381041538be6c92858d6ba13 from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:55.707135 850 leader_election.cc:343] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:55.705065 1157 ts_tablet_manager.cc:1939] T e962a020664b42ea918e5952dc43aade P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.707556 1156 ts_tablet_manager.cc:1918] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.708747 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e962a020664b42ea918e5952dc43aade (table default.Table_e0533c0c40ca4af1be6ac30ee37f3aed [id=1d5fd1e402a54d31b3fea01d38db6918]) successfully deleted
W20250629 01:51:55.709391 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 60e12d28ef784d898dbf9846f7063fdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60e12d28ef784d898dbf9846f7063fdf already in progress: creating tablet
W20250629 01:51:55.710047 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e332a512574e4b4e881aeb37e289f0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e332a512574e4b4e881aeb37e289f0bc already in progress: creating tablet
W20250629 01:51:55.710609 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a53d9e8b7efe46099821ae628640a42e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53d9e8b7efe46099821ae628640a42e already in progress: creating tablet
I20250629 01:51:55.711380 1156 log.cc:1199] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/6f3a65c9c1c740b1865b96a6e2aea524
W20250629 01:51:55.711364 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e5d27d658892496a9b19b57949a8399a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5d27d658892496a9b19b57949a8399a already in progress: creating tablet
I20250629 01:51:55.711743 1156 ts_tablet_manager.cc:1939] T 6f3a65c9c1c740b1865b96a6e2aea524 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:55.712069 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ac560295fddd44368e6f9bc543f7cbfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac560295fddd44368e6f9bc543f7cbfa already in progress: creating tablet
I20250629 01:51:55.712476 1157 tablet_replica.cc:331] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.712613 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1000) took 1451 ms. Trace:
I20250629 01:51:55.712790 1157 raft_consensus.cc:2241] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.713076 1157 raft_consensus.cc:2270] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:55.717636 850 leader_election.cc:343] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:55.717731 850 leader_election.cc:304] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.717886 1156 rpcz_store.cc:276] 0629 01:51:54.261448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.261781 (+ 333us) service_pool.cc:225] Handling call
0629 01:51:55.689630 (+1427849us) ts_tablet_manager.cc:1145] Deleting tablet 6f3a65c9c1c740b1865b96a6e2aea524
0629 01:51:55.689637 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.693251 (+ 3614us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.707489 (+ 14238us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:55.712598 (+ 5109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1427676}
I20250629 01:51:55.717994 2271 tablet.cc:1722] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:55.718071 2217 raft_consensus.cc:2747] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.717842 1155 tablet_bootstrap.cc:492] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.718295 1155 ts_tablet_manager.cc:1397] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.008s
I20250629 01:51:55.718477 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 6f3a65c9c1c740b1865b96a6e2aea524 (table default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]) successfully deleted
I20250629 01:51:55.718927 455 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:55.719305 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:55.719475 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:51:55.719648 448 catalog_manager.cc:5582] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:55.719991 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]). Expected version 1 got 0
I20250629 01:51:55.720022 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9f59fbe9634ac8b27f990d9496f637" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.720271 1052 raft_consensus.cc:2466] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:51:55.720692 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 303f77a729c648f9b1146d11056b242d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303f77a729c648f9b1146d11056b242d already in progress: creating tablet
I20250629 01:51:55.720854 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
W20250629 01:51:55.721045 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 3ba5887dffc1460a89208447ecc6a198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ba5887dffc1460a89208447ecc6a198 already in progress: creating tablet
W20250629 01:51:55.721354 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2521a476643c41c1a2e5282755a58091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2521a476643c41c1a2e5282755a58091 already in progress: creating tablet
W20250629 01:51:55.721642 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a2e9de48e4ad401880f351f2bd55cf5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e9de48e4ad401880f351f2bd55cf5c already in progress: creating tablet
I20250629 01:51:55.722110 1155 raft_consensus.cc:357] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:55.722180 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9ed1ee73bb8f4761806a66f988b57bf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed1ee73bb8f4761806a66f988b57bf9 already in progress: creating tablet
I20250629 01:51:55.722234 1155 raft_consensus.cc:383] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.722272 1155 raft_consensus.cc:738] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.722743 1155 consensus_queue.cc:260] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.723757 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:55.724880 1155 ts_tablet_manager.cc:1428] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:55.725134 1155 tablet_bootstrap.cc:492] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.726532 1155 tablet_bootstrap.cc:654] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.728652 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1001) took 1443 ms. Trace:
I20250629 01:51:55.729485 1156 rpcz_store.cc:276] 0629 01:51:54.285483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.286857 (+ 1374us) service_pool.cc:225] Handling call
0629 01:51:55.728537 (+1441680us) ts_tablet_manager.cc:1145] Deleting tablet ce7a72db953646038a9d82dde04d76e4
0629 01:51:55.728548 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.728633 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1441517}
I20250629 01:51:55.730010 710 leader_election.cc:304] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.730247 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1002) took 1410 ms. Trace:
I20250629 01:51:55.730329 2049 raft_consensus.cc:2802] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:55.730427 2049 raft_consensus.cc:695] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:55.730590 2049 consensus_queue.cc:237] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.731437 881 tablet_service.cc:1468] Processing CreateTablet for tablet d54bdcf7e1a14c669472af8acf36aac2 (DEFAULT_TABLE table=default.Table_02f34269d06f4dfb9d03d3da0aea953d [id=b68067ef1fcf484ba89eb9634de9aeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.731840 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54bdcf7e1a14c669472af8acf36aac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.732846 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:51:55.733050 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:55.734131 745 tablet_service.cc:1515] Processing DeleteTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.735013 1159 tablet_bootstrap.cc:492] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:55.735110 1159 ts_tablet_manager.cc:1397] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.002s
I20250629 01:51:55.735355 2217 raft_consensus.cc:491] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.735450 2217 raft_consensus.cc:513] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } }
I20250629 01:51:55.735903 2217 leader_election.cc:290] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:55.736721 850 leader_election.cc:343] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 683c87ea2ad44cecb2366d4a846d9195
W20250629 01:51:55.736886 850 leader_election.cc:343] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 683c87ea2ad44cecb2366d4a846d9195
I20250629 01:51:55.736965 850 leader_election.cc:304] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.737342 2217 raft_consensus.cc:2747] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.730337 1156 rpcz_store.cc:276] 0629 01:51:54.320194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.320241 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:55.730156 (+1409915us) ts_tablet_manager.cc:1145] Deleting tablet f4cf919a96f34167b7dc9c3dfe7946a4
0629 01:51:55.730164 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.730232 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409769}
W20250629 01:51:55.737960 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f4cf919a96f34167b7dc9c3dfe7946a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4cf919a96f34167b7dc9c3dfe7946a4 already in progress: creating tablet
I20250629 01:51:55.738306 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1003) took 1407 ms. Trace:
I20250629 01:51:55.738256 1159 raft_consensus.cc:357] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.738376 1156 rpcz_store.cc:276] 0629 01:51:54.331084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.331260 (+ 176us) service_pool.cc:225] Handling call
0629 01:51:55.738218 (+1406958us) ts_tablet_manager.cc:1145] Deleting tablet cfb025535ecd4cf88681c4951a56ba29
0629 01:51:55.738226 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.738292 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406735}
I20250629 01:51:55.738587 1159 raft_consensus.cc:383] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.738754 1159 raft_consensus.cc:738] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.739171 1159 consensus_queue.cc:260] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:55.739398 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet cfb025535ecd4cf88681c4951a56ba29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfb025535ecd4cf88681c4951a56ba29 already in progress: creating tablet
I20250629 01:51:55.739449 1156 tablet_replica.cc:331] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:55.739590 1156 raft_consensus.cc:2241] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.740296 1156 raft_consensus.cc:2270] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.740962 1156 ts_tablet_manager.cc:1905] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.741429 745 tablet_service.cc:1468] Processing CreateTablet for tablet d54bdcf7e1a14c669472af8acf36aac2 (DEFAULT_TABLE table=default.Table_02f34269d06f4dfb9d03d3da0aea953d [id=b68067ef1fcf484ba89eb9634de9aeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.741957 881 tablet_service.cc:1468] Processing CreateTablet for tablet f7e6f33c7bee44d1a2b6feb0625faedf (DEFAULT_TABLE table=default.Table_3cd6904b29dc4d52bb6acb613babf904 [id=259db841e6904f4faddfafd0a0e63a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.742229 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54bdcf7e1a14c669472af8acf36aac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.743594 889 tablet_service.cc:1468] Processing CreateTablet for tablet cde5df0a015744bf804d74c8046ed458 (DEFAULT_TABLE table=default.Table_db4b4ae9a38f4d75845beeffd7251609 [id=a01b6cb4373347fabb48bf53bc9ccd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.744521 740 tablet_service.cc:1468] Processing CreateTablet for tablet cde5df0a015744bf804d74c8046ed458 (DEFAULT_TABLE table=default.Table_db4b4ae9a38f4d75845beeffd7251609 [id=a01b6cb4373347fabb48bf53bc9ccd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.744823 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde5df0a015744bf804d74c8046ed458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.745745 2238 tablet.cc:1717] T 0323947b135a4308903b1eed1ca4db71 P ef5c5979bb9e475f983af7c3e9e0e019: Skipping requested alter to schema version 3, tablet already version 3
I20250629 01:51:55.746992 710 leader_election.cc:304] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.747300 1861 raft_consensus.cc:2802] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:55.747388 1861 raft_consensus.cc:695] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:55.747359 754 tablet_service.cc:1468] Processing CreateTablet for tablet f7e6f33c7bee44d1a2b6feb0625faedf (DEFAULT_TABLE table=default.Table_3cd6904b29dc4d52bb6acb613babf904 [id=259db841e6904f4faddfafd0a0e63a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.747643 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7e6f33c7bee44d1a2b6feb0625faedf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.748420 710 leader_election.cc:343] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 52be184d0480495a894442d6f72a0f12
I20250629 01:51:55.748509 710 leader_election.cc:304] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:55.748759 2052 raft_consensus.cc:2747] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.749207 710 leader_election.cc:304] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:55.749265 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde5df0a015744bf804d74c8046ed458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.749491 2052 raft_consensus.cc:2802] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:55.749547 2052 raft_consensus.cc:491] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:55.749595 2052 raft_consensus.cc:3058] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.750671 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 8799b1858bf2408a85f0956578639393 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:43 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.751667 2052 raft_consensus.cc:513] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.752327 2052 leader_election.cc:290] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.752511 888 tablet_service.cc:1468] Processing CreateTablet for tablet 23d539ff02dd4f2ca5a8f070276bf901 (DEFAULT_TABLE table=default.Table_c3c28291b8e046aebf0a8accc24e0599 [id=e7265d794ecc467596c30273910dd873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.752754 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d539ff02dd4f2ca5a8f070276bf901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.754153 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7e6f33c7bee44d1a2b6feb0625faedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.754356 889 tablet_service.cc:1468] Processing CreateTablet for tablet 77727efeef5e4f0c89f4afca2941d02a (DEFAULT_TABLE table=default.Table_45ad1da33f23408c858cd346e7ac71bf [id=b59ee7ba9aba4cd1b61e32be0ab1dce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.754611 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77727efeef5e4f0c89f4afca2941d02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.756269 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9f59fbe9634ac8b27f990d9496f637" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:55.756546 1159 ts_tablet_manager.cc:1428] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20250629 01:51:55.756639 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8cba33eba4894752996a32bcefa3bd2d (DEFAULT_TABLE table=default.Table_2a7ba606f3fe4a2b8233338d373dd389 [id=324287a932d646e6ae20f0c17defd1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.756422 2052 consensus_queue.cc:1035] T 1bb4ff3220cf4d25b680235f1b4882ed P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:55.756883 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cba33eba4894752996a32bcefa3bd2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.757503 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f5bc90b9f24640639b11b63857131036" } modify_external_catalogs: true
I20250629 01:51:55.757635 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f5bc90b9f24640639b11b63857131036" } modify_external_catalogs: true
I20250629 01:51:55.757983 1159 tablet_bootstrap.cc:492] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:55.713799 1157 ts_tablet_manager.cc:1905] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.747570 1861 consensus_queue.cc:237] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:55.761112 712 leader_election.cc:343] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.761498 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1602ef35993a40909b316edb27f35b40" } modify_external_catalogs: true
I20250629 01:51:55.761647 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1602ef35993a40909b316edb27f35b40" } modify_external_catalogs: true
I20250629 01:51:55.762452 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8cba33eba4894752996a32bcefa3bd2d (DEFAULT_TABLE table=default.Table_2a7ba606f3fe4a2b8233338d373dd389 [id=324287a932d646e6ae20f0c17defd1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.762742 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cba33eba4894752996a32bcefa3bd2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.764252 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c08e69cfa0894f67beafee56aa098ad9" } modify_external_catalogs: true
I20250629 01:51:55.764391 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c08e69cfa0894f67beafee56aa098ad9" } modify_external_catalogs: true
I20250629 01:51:55.765810 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c456f245c84847adb29db0e98c23bde7" } modify_external_catalogs: true
I20250629 01:51:55.765926 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c456f245c84847adb29db0e98c23bde7" } modify_external_catalogs: true
I20250629 01:51:55.766013 1159 tablet_bootstrap.cc:654] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.767326 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_02f34269d06f4dfb9d03d3da0aea953d" } modify_external_catalogs: true
I20250629 01:51:55.767450 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_02f34269d06f4dfb9d03d3da0aea953d" } modify_external_catalogs: true
I20250629 01:51:55.767939 740 tablet_service.cc:1468] Processing CreateTablet for tablet 23d539ff02dd4f2ca5a8f070276bf901 (DEFAULT_TABLE table=default.Table_c3c28291b8e046aebf0a8accc24e0599 [id=e7265d794ecc467596c30273910dd873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.768509 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d539ff02dd4f2ca5a8f070276bf901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.772643 2273 raft_consensus.cc:491] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.772782 2273 raft_consensus.cc:513] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.773314 2273 leader_election.cc:290] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.774304 2271 tablet.cc:1717] T 0323947b135a4308903b1eed1ca4db71 P 4712643b602e4f898752fcab6a1e5913: Skipping requested alter to schema version 3, tablet already version 3
I20250629 01:51:55.775089 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d6bf3c7d0bb548e0b77ba8b569a72d0d
I20250629 01:51:55.776327 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347fa6a12a744ee88058643b7143e39d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:55.776810 986 leader_election.cc:343] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.777310 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347fa6a12a744ee88058643b7143e39d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.777828 988 leader_election.cc:343] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.777942 988 leader_election.cc:304] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.778311 2273 raft_consensus.cc:2747] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.778645 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9f59fbe9634ac8b27f990d9496f637" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:55.778826 1052 raft_consensus.cc:3058] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.778816 754 tablet_service.cc:1468] Processing CreateTablet for tablet 77727efeef5e4f0c89f4afca2941d02a (DEFAULT_TABLE table=default.Table_45ad1da33f23408c858cd346e7ac71bf [id=b59ee7ba9aba4cd1b61e32be0ab1dce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.779211 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77727efeef5e4f0c89f4afca2941d02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.779381 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet add482b533d045e7a3010064fae730cf
I20250629 01:51:55.780019 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 11b4bdbbf5d3421ab6e3a31de90fff6d
I20250629 01:51:55.780375 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 556c2050d2cb40f28a3a0bfb9d677d69
I20250629 01:51:55.781253 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d54bdcf7e1a14c669472af8acf36aac2
I20250629 01:51:55.781349 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.781428 888 tablet_service.cc:1515] Processing DeleteTablet for tablet add482b533d045e7a3010064fae730cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.781514 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 556c2050d2cb40f28a3a0bfb9d677d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.781955 738 tablet_service.cc:1515] Processing DeleteTablet for tablet d54bdcf7e1a14c669472af8acf36aac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.782135 749 tablet_service.cc:1515] Processing DeleteTablet for tablet add482b533d045e7a3010064fae730cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.781355 745 tablet_service.cc:1515] Processing DeleteTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:55.782364 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.782514 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 556c2050d2cb40f28a3a0bfb9d677d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.782655 888 tablet_service.cc:1515] Processing DeleteTablet for tablet d54bdcf7e1a14c669472af8acf36aac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.782732 1052 raft_consensus.cc:2466] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:55.783725 448 catalog_manager.cc:5582] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:55.786293 889 tablet_service.cc:1515] Processing DeleteTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:55.788288 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e5e8b8e4d42f43c78f34102b76b4b11a (table default.Table_904b8b7a3ab1414a9bd29274648bbb86 [id=e991e29b3d9d4dcc96912754565336f8]). Expected version 1 got 0
I20250629 01:51:55.790366 754 tablet_service.cc:1468] Processing CreateTablet for tablet b1ca655a041b4aff95df14041a51b6cf (DEFAULT_TABLE table=default.Table_e4309edc8dd64fc88947c246a80faac7 [id=2df5022bc3884ebcb91ba7353c5b3bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.790663 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ca655a041b4aff95df14041a51b6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.792207 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 95247cc778504f4d85d9013f84c59e69 (table default.Table_355a96774d4f4e0e9f5835f7a7450e0c [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]). Expected version 1 got 0
I20250629 01:51:55.795006 2049 raft_consensus.cc:491] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.795161 2049 raft_consensus.cc:513] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.795873 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0b9afb9976541b2bc7ab188f35a62ec" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:55.795938 2049 leader_election.cc:290] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:55.796272 712 leader_election.cc:343] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.796430 889 tablet_service.cc:1468] Processing CreateTablet for tablet b1ca655a041b4aff95df14041a51b6cf (DEFAULT_TABLE table=default.Table_e4309edc8dd64fc88947c246a80faac7 [id=2df5022bc3884ebcb91ba7353c5b3bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.796615 2049 raft_consensus.cc:491] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.796718 2049 raft_consensus.cc:513] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.797317 2049 leader_election.cc:290] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.797375 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ca655a041b4aff95df14041a51b6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.797606 740 tablet_service.cc:1468] Processing CreateTablet for tablet 18df997317c04431a496448a4836ac2a (DEFAULT_TABLE table=default.Table_4356a70d10bd41e4ad6b77e5612a81ef [id=50b83fb9661b4e0f9ea0e5003ba63b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.797644 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.798017 712 leader_election.cc:343] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.799861 888 tablet_service.cc:1468] Processing CreateTablet for tablet 18df997317c04431a496448a4836ac2a (DEFAULT_TABLE table=default.Table_4356a70d10bd41e4ad6b77e5612a81ef [id=50b83fb9661b4e0f9ea0e5003ba63b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.800204 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18df997317c04431a496448a4836ac2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.801410 745 tablet_service.cc:1468] Processing CreateTablet for tablet 1a05cd13ca124ff399e10b4ca79b6ccf (DEFAULT_TABLE table=default.Table_5257d4d9f670497287003f5c8bf98456 [id=4266d70c7a514dc8af01519db5002340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.803915 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18df997317c04431a496448a4836ac2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.804073 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a05cd13ca124ff399e10b4ca79b6ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.806063 881 tablet_service.cc:1468] Processing CreateTablet for tablet 1a05cd13ca124ff399e10b4ca79b6ccf (DEFAULT_TABLE table=default.Table_5257d4d9f670497287003f5c8bf98456 [id=4266d70c7a514dc8af01519db5002340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.806413 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a05cd13ca124ff399e10b4ca79b6ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.808552 2273 raft_consensus.cc:491] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.808655 2273 raft_consensus.cc:513] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.809124 2273 leader_election.cc:290] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.809614 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:55.809751 775 raft_consensus.cc:2466] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:55.810091 986 leader_election.cc:304] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:55.810279 2273 raft_consensus.cc:2802] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:55.810325 2273 raft_consensus.cc:491] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:55.810369 2273 raft_consensus.cc:3058] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.795045 1155 tablet_bootstrap.cc:492] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.812208 1155 ts_tablet_manager.cc:1397] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.005s
I20250629 01:51:55.812553 2273 raft_consensus.cc:513] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.812812 1155 raft_consensus.cc:357] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.812950 1155 raft_consensus.cc:383] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.813000 1155 raft_consensus.cc:738] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.813143 1155 consensus_queue.cc:260] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.813206 2273 leader_election.cc:290] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.813601 745 tablet_service.cc:1468] Processing CreateTablet for tablet 60c79b5f0c1a4a9b82c9f57f43cf2a64 (DEFAULT_TABLE table=default.Table_02190f4ee1a746e7b458b005042fbc0f [id=b5367f695a8645a4896ab11f648bcec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.813786 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:55.813905 775 raft_consensus.cc:3058] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.815594 749 tablet_service.cc:1468] Processing CreateTablet for tablet feeaa50e7f224da69128ea916e4145fd (DEFAULT_TABLE table=default.Table_e646217c593b412f8d30f34145ea5325 [id=2b6021fe8aef44e7a806e56489e34c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.816470 775 raft_consensus.cc:2466] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:55.816803 986 leader_election.cc:304] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:55.817476 896 tablet_service.cc:1468] Processing CreateTablet for tablet feeaa50e7f224da69128ea916e4145fd (DEFAULT_TABLE table=default.Table_e646217c593b412f8d30f34145ea5325 [id=2b6021fe8aef44e7a806e56489e34c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.817534 2273 raft_consensus.cc:2802] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:55.817554 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:55.817729 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feeaa50e7f224da69128ea916e4145fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.817760 2273 raft_consensus.cc:695] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:55.817974 2273 consensus_queue.cc:237] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:55.819322 988 leader_election.cc:343] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.819415 883 tablet_service.cc:1468] Processing CreateTablet for tablet ba9b705b1f8c4292ab33bc47aab3ca12 (DEFAULT_TABLE table=default.Table_85e8350afe3f4983b2225325b3c58208 [id=837588cbd00f48caafa3c004e2bd852b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.819653 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9b705b1f8c4292ab33bc47aab3ca12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.820204 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.821153 988 leader_election.cc:343] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.822937 448 catalog_manager.cc:5582] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:55.823364 738 tablet_service.cc:1468] Processing CreateTablet for tablet ba9b705b1f8c4292ab33bc47aab3ca12 (DEFAULT_TABLE table=default.Table_85e8350afe3f4983b2225325b3c58208 [id=837588cbd00f48caafa3c004e2bd852b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.824254 2217 raft_consensus.cc:491] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.824362 2217 raft_consensus.cc:513] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.824813 2217 leader_election.cc:290] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.825822 883 tablet_service.cc:1468] Processing CreateTablet for tablet 89aa7f502b1a4198aea2096105dd00d6 (DEFAULT_TABLE table=default.Table_3c3ba33385624970b8f09248b76e5cc1 [id=d34b612ad43d4067869c532a99f86d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.826087 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89aa7f502b1a4198aea2096105dd00d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.826543 1155 ts_tablet_manager.cc:1428] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250629 01:51:55.827252 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "013ab393a0f14bdc89389349bcc5d24b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:55.825822 890 tablet_service.cc:1468] Processing CreateTablet for tablet 60c79b5f0c1a4a9b82c9f57f43cf2a64 (DEFAULT_TABLE table=default.Table_02190f4ee1a746e7b458b005042fbc0f [id=b5367f695a8645a4896ab11f648bcec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.828598 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c79b5f0c1a4a9b82c9f57f43cf2a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.828765 1155 tablet_bootstrap.cc:492] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.830346 754 tablet_service.cc:1468] Processing CreateTablet for tablet 89aa7f502b1a4198aea2096105dd00d6 (DEFAULT_TABLE table=default.Table_3c3ba33385624970b8f09248b76e5cc1 [id=d34b612ad43d4067869c532a99f86d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.831252 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "013ab393a0f14bdc89389349bcc5d24b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.813915 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c79b5f0c1a4a9b82c9f57f43cf2a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.832393 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9b705b1f8c4292ab33bc47aab3ca12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.832731 913 raft_consensus.cc:3058] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:55.833783 1155 tablet_bootstrap.cc:654] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.836802 850 leader_election.cc:304] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.836938 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.837262 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0b9afb9976541b2bc7ab188f35a62ec" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:55.837425 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feeaa50e7f224da69128ea916e4145fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.837877 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 8cba33eba4894752996a32bcefa3bd2d (DEFAULT_TABLE table=default.Table_2a7ba606f3fe4a2b8233338d373dd389 [id=324287a932d646e6ae20f0c17defd1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.838146 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cba33eba4894752996a32bcefa3bd2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.840934 2217 raft_consensus.cc:2747] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.841976 1030 tablet_service.cc:1468] Processing CreateTablet for tablet cde5df0a015744bf804d74c8046ed458 (DEFAULT_TABLE table=default.Table_db4b4ae9a38f4d75845beeffd7251609 [id=a01b6cb4373347fabb48bf53bc9ccd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.842283 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cde5df0a015744bf804d74c8046ed458. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.844118 1861 raft_consensus.cc:491] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.844224 1861 raft_consensus.cc:513] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.844887 1861 leader_election.cc:290] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.845314 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2aca26af31ed4bc994f4d62603edc881" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:55.845634 712 leader_election.cc:343] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:55.847589 1155 tablet_bootstrap.cc:492] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.847705 1155 ts_tablet_manager.cc:1397] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20250629 01:51:55.847980 1157 ts_tablet_manager.cc:1918] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.848132 1157 log.cc:1199] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/47d4287317ec4788bd96ad174c31748d
I20250629 01:51:55.848452 1155 raft_consensus.cc:357] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.848528 1157 ts_tablet_manager.cc:1939] T 47d4287317ec4788bd96ad174c31748d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:55.848596 1155 raft_consensus.cc:383] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.848646 1155 raft_consensus.cc:738] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.848821 1155 consensus_queue.cc:260] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.849047 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d54bdcf7e1a14c669472af8acf36aac2 (DEFAULT_TABLE table=default.Table_02f34269d06f4dfb9d03d3da0aea953d [id=b68067ef1fcf484ba89eb9634de9aeb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.849222 1155 ts_tablet_manager.cc:1428] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:55.849381 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d54bdcf7e1a14c669472af8acf36aac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.849450 1155 tablet_bootstrap.cc:492] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.850697 1155 tablet_bootstrap.cc:654] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.852270 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89aa7f502b1a4198aea2096105dd00d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.858557 1154 tablet_bootstrap.cc:492] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.858668 1154 ts_tablet_manager.cc:1397] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.213s user 0.000s sys 0.026s
I20250629 01:51:55.859601 1154 raft_consensus.cc:357] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.859755 1154 raft_consensus.cc:383] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.859808 1154 raft_consensus.cc:738] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.859956 1154 consensus_queue.cc:260] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.860455 1154 ts_tablet_manager.cc:1428] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:51:55.861172 1154 tablet_bootstrap.cc:492] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.862490 1154 tablet_bootstrap.cc:654] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.864334 1157 tablet_replica.cc:331] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:55.864500 740 tablet_service.cc:1468] Processing CreateTablet for tablet 661f4bf68fc14cfdaa86ff86abc8ab74 (DEFAULT_TABLE table=default.Table_3efca07702364cf982f01059b4d3899c [id=3aa699feb56c4af5b4052779edd43bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.864585 1157 raft_consensus.cc:2241] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.864785 1157 raft_consensus.cc:2270] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.864835 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 661f4bf68fc14cfdaa86ff86abc8ab74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.865466 1157 ts_tablet_manager.cc:1905] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.865525 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7aef2514a87c4c4aaf4d51b314630185 (DEFAULT_TABLE table=default.Table_a1e8b8763bff46768cb4e15ba61d470b [id=9d8c7b07069d4b86b2a5e18fa3e88ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.865804 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aef2514a87c4c4aaf4d51b314630185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.874971 1155 tablet_bootstrap.cc:492] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:55.875097 1155 ts_tablet_manager.cc:1397] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:51:55.875622 1155 raft_consensus.cc:357] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.875738 1155 raft_consensus.cc:383] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.875775 1155 raft_consensus.cc:738] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.878213 1155 consensus_queue.cc:260] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.878587 1155 ts_tablet_manager.cc:1428] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:55.878865 745 tablet_service.cc:1468] Processing CreateTablet for tablet 999a62f27eb441699f641562ae60a287 (DEFAULT_TABLE table=default.Table_6e07ccb977954917826f8f245c6be17a [id=5a8e712d2cec42cf812be7aa2302e4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.881038 749 tablet_service.cc:1468] Processing CreateTablet for tablet 8f56d625aa5e456d84571b15184e1807 (DEFAULT_TABLE table=default.Table_b296280dfbe1428b9e98ebbb01dacc44 [id=cf990cf769874bd88531d52c02f22a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.883957 448 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_f798e82e14704a689fe84cbeb9a8d477 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 4 got 0
I20250629 01:51:55.889200 738 tablet_service.cc:1468] Processing CreateTablet for tablet b6d60a3c1bc642969ced8c712e5af1c1 (DEFAULT_TABLE table=default.Table_20c6268f43c64a5480574ad87d428a7c [id=1b897d7a629e4a26998e86e4f1ebb33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.833001 451 catalog_manager.cc:5582] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:55.840543 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:55.889989 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_45100053e91845d1b945caac267d7349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.890408 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45100053e91845d1b945caac267d7349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.890614 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4a345f9896144379bc0d923ac517a0ed (DEFAULT_TABLE table=default.Table_716fe511fb2c4fb384a34d9964136cab [id=4492edac31974b02900902435520113b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.849642 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 47d4287317ec4788bd96ad174c31748d (table default.Table_58a60b60dab64affa9d8b5b1acf53326 [id=d524cd9f7aba4981b939a770099325c8]) successfully deleted
I20250629 01:51:55.891759 750 tablet_service.cc:1468] Processing CreateTablet for tablet c7806330425045ed809f092f20ada6fd (DEFAULT_TABLE table=default.Table_551758fbd6ed485096040f978cfec8af [id=3594a358e67e46edbff8113ca4ea7e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.892699 896 tablet_service.cc:1468] Processing CreateTablet for tablet b6d60a3c1bc642969ced8c712e5af1c1 (DEFAULT_TABLE table=default.Table_20c6268f43c64a5480574ad87d428a7c [id=1b897d7a629e4a26998e86e4f1ebb33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.892802 879 tablet_service.cc:1468] Processing CreateTablet for tablet c7806330425045ed809f092f20ada6fd (DEFAULT_TABLE table=default.Table_551758fbd6ed485096040f978cfec8af [id=3594a358e67e46edbff8113ca4ea7e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.893174 895 tablet_service.cc:1468] Processing CreateTablet for tablet 4a345f9896144379bc0d923ac517a0ed (DEFAULT_TABLE table=default.Table_716fe511fb2c4fb384a34d9964136cab [id=4492edac31974b02900902435520113b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.859072 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a768996f666e43669ed0128c8dde5429" } new_table_name: "default.Table_aca8c2c6ca5045838333bfd1e7d46d50" modify_external_catalogs: true
I20250629 01:51:55.893493 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d60a3c1bc642969ced8c712e5af1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.859470 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b913ec7dcfc14a73ae38f883f9e5296d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.894819 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b913ec7dcfc14a73ae38f883f9e5296d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.895195 740 tablet_service.cc:1468] Processing CreateTablet for tablet 8be7d65065fa4ed6956cad11d2cb5ffb (DEFAULT_TABLE table=default.Table_29b731f4ff984ce5bb98b903fbb804c0 [id=26d79160a7d94ba1a1dd4c0e25d52278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.861403 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8f370719dbf6400987554c20446591bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.895853 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f370719dbf6400987554c20446591bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.871583 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_90112469ff66420a85ce68ec4cc69b56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.896839 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90112469ff66420a85ce68ec4cc69b56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.872124 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e50ddf6a53544cd59126c103fb46128f" } new_table_name: "default.Table_4ae5ee4f0ebe46d7beb3958a50f0dd87" modify_external_catalogs: true
I20250629 01:51:55.874361 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_503924285343420d8e275498070e193b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.897943 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_503924285343420d8e275498070e193b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.874904 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d7fd94b003704c088b36cb0df05bd7f5" } new_table_name: "default.Table_eb16c73366544d7e8255f63a85518f0f" modify_external_catalogs: true
I20250629 01:51:55.899948 1154 tablet_bootstrap.cc:492] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.900048 1154 ts_tablet_manager.cc:1397] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.001s
I20250629 01:51:55.900210 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7806330425045ed809f092f20ada6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.900708 1154 raft_consensus.cc:357] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.900854 1154 raft_consensus.cc:383] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.900905 1154 raft_consensus.cc:738] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.901094 1154 consensus_queue.cc:260] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.901494 1154 ts_tablet_manager.cc:1428] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:55.901710 1154 tablet_bootstrap.cc:492] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.903055 1154 tablet_bootstrap.cc:654] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.903618 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a345f9896144379bc0d923ac517a0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.904847 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2818fd021c504a3aa687ad16bf8a0c48 (DEFAULT_TABLE table=default.Table_d5042d6281a54e70861a5619a2967981 [id=e050eacfc013416fad52f49422828078]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.905143 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2818fd021c504a3aa687ad16bf8a0c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.905968 884 tablet_service.cc:1468] Processing CreateTablet for tablet 53c2c00c5f7b4cefb88523b8f5453c7e (DEFAULT_TABLE table=default.Table_a0699bfad44f4661b365b0924fd37513 [id=c42f1afbcc8e45fda61e26b602b5e89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.906224 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c2c00c5f7b4cefb88523b8f5453c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.907967 2217 raft_consensus.cc:491] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.908070 2217 raft_consensus.cc:513] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.908136 878 tablet_service.cc:1468] Processing CreateTablet for tablet b7e50eb762144fc28c531af7514776b0 (DEFAULT_TABLE table=default.Table_6f23d8816c084fafa5c57899912ffdd6 [id=c52afe871a3d49e980d364ac0e39d14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.908365 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e50eb762144fc28c531af7514776b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.908564 2217 leader_election.cc:290] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:55.909605 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:55.910476 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a526befbb9d14f6388686ec1248aa292" } new_table_name: "default.Table_121b994e3a964726b696940dae054ccd" modify_external_catalogs: true
I20250629 01:51:55.911953 741 tablet_service.cc:1468] Processing CreateTablet for tablet 2818fd021c504a3aa687ad16bf8a0c48 (DEFAULT_TABLE table=default.Table_d5042d6281a54e70861a5619a2967981 [id=e050eacfc013416fad52f49422828078]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.912001 884 tablet_service.cc:1468] Processing CreateTablet for tablet 631ca7239f574441b37c11e104c3d56b (DEFAULT_TABLE table=default.Table_39a0ff1dd8064dc4a2990b95fac35099 [id=811dcd76b5c742f69d611cce6f6bde86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.912273 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631ca7239f574441b37c11e104c3d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.912286 739 tablet_service.cc:1468] Processing CreateTablet for tablet b7e50eb762144fc28c531af7514776b0 (DEFAULT_TABLE table=default.Table_6f23d8816c084fafa5c57899912ffdd6 [id=c52afe871a3d49e980d364ac0e39d14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.912595 757 tablet_service.cc:1468] Processing CreateTablet for tablet 53c2c00c5f7b4cefb88523b8f5453c7e (DEFAULT_TABLE table=default.Table_a0699bfad44f4661b365b0924fd37513 [id=c42f1afbcc8e45fda61e26b602b5e89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.914654 888 tablet_service.cc:1468] Processing CreateTablet for tablet 8be7d65065fa4ed6956cad11d2cb5ffb (DEFAULT_TABLE table=default.Table_29b731f4ff984ce5bb98b903fbb804c0 [id=26d79160a7d94ba1a1dd4c0e25d52278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.914934 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8be7d65065fa4ed6956cad11d2cb5ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.916671 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9d377064eeb847699a17f707f0cae83c" } new_table_name: "default.Table_b0c3b681591d46c79424b1e3ee842cf1" modify_external_catalogs: true
I20250629 01:51:55.917867 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d5e5c0a7d4814ee6895122e2eb2a0717"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.918265 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5e5c0a7d4814ee6895122e2eb2a0717 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.918766 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7aef2514a87c4c4aaf4d51b314630185 (DEFAULT_TABLE table=default.Table_a1e8b8763bff46768cb4e15ba61d470b [id=9d8c7b07069d4b86b2a5e18fa3e88ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.919093 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aef2514a87c4c4aaf4d51b314630185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.919090 884 tablet_service.cc:1468] Processing CreateTablet for tablet ad1ba1fec9bf49a5a44a193802edde0a (DEFAULT_TABLE table=default.Table_efd88358bbde43528c5d3013bb7aaaf2 [id=05c1626a711d405996ae1c40256e4414]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:55.919489 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbe1f8083a884cdd97afce83bd22073c consensus state has no leader
I20250629 01:51:55.919607 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1ba1fec9bf49a5a44a193802edde0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.919764 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d651acfdb0eb4c21a5301d4b4acd35e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.920161 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d651acfdb0eb4c21a5301d4b4acd35e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.921249 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_739fd1ba816f484ab8a493fe8703336f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:55.921622 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_739fd1ba816f484ab8a493fe8703336f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:55.922365 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8bed6b19b8454766add0f458a1eac7c4 (DEFAULT_TABLE table=default.Table_adcbb5e4f5cb4ae78e1f48f042c4033e [id=3391dd3dacc2450fbb71e469e866ad78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.922597 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bed6b19b8454766add0f458a1eac7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.925158 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8f56d625aa5e456d84571b15184e1807 (DEFAULT_TABLE table=default.Table_b296280dfbe1428b9e98ebbb01dacc44 [id=cf990cf769874bd88531d52c02f22a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.925491 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f56d625aa5e456d84571b15184e1807. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.925943 742 tablet_service.cc:1468] Processing CreateTablet for tablet 631ca7239f574441b37c11e104c3d56b (DEFAULT_TABLE table=default.Table_39a0ff1dd8064dc4a2990b95fac35099 [id=811dcd76b5c742f69d611cce6f6bde86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.926085 743 tablet_service.cc:1468] Processing CreateTablet for tablet ad1ba1fec9bf49a5a44a193802edde0a (DEFAULT_TABLE table=default.Table_efd88358bbde43528c5d3013bb7aaaf2 [id=05c1626a711d405996ae1c40256e4414]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.926266 746 tablet_service.cc:1468] Processing CreateTablet for tablet 12d63924c5ff4a99ac62f89a9d29423b (DEFAULT_TABLE table=default.Table_dae51ebf34aa422a9e59b9369d5564a0 [id=f064321537ad49aeab2f63a840b16764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.926550 751 tablet_service.cc:1468] Processing CreateTablet for tablet 8bed6b19b8454766add0f458a1eac7c4 (DEFAULT_TABLE table=default.Table_adcbb5e4f5cb4ae78e1f48f042c4033e [id=3391dd3dacc2450fbb71e469e866ad78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.927446 1154 tablet_bootstrap.cc:492] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:55.927546 1154 ts_tablet_manager.cc:1397] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:51:55.928187 1154 raft_consensus.cc:357] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.928323 1154 raft_consensus.cc:383] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:55.928378 1154 raft_consensus.cc:738] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:55.928567 1154 consensus_queue.cc:260] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:55.929062 1154 ts_tablet_manager.cc:1428] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:51:55.929314 1154 tablet_bootstrap.cc:492] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:55.930609 1154 tablet_bootstrap.cc:654] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.931782 883 tablet_service.cc:1468] Processing CreateTablet for tablet 661f4bf68fc14cfdaa86ff86abc8ab74 (DEFAULT_TABLE table=default.Table_3efca07702364cf982f01059b4d3899c [id=3aa699feb56c4af5b4052779edd43bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.932406 747 tablet_service.cc:1468] Processing CreateTablet for tablet d2870ea06d364d2cb3a00d428b831df3 (DEFAULT_TABLE table=default.Table_254111a6f5a74f2c9f3b57c27da3b8a3 [id=979ea777ca4349cdbf0291c16b1723f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.934181 744 tablet_service.cc:1468] Processing CreateTablet for tablet 5552fced2b794ed5b219de28b2c307cf (DEFAULT_TABLE table=default.Table_d0e7545edb1e4ed1b67d3167c05b0995 [id=02bde06a6b2940a9be983b400783223c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.934334 886 tablet_service.cc:1468] Processing CreateTablet for tablet 46bb76c63ce749078365420f224d62f3 (DEFAULT_TABLE table=default.Table_825efeee6983439b9a45a82092f1e29e [id=ea2235e61e034f0c823898696bbec076]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:55.935676 850 leader_election.cc:343] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:55.936496 752 tablet_service.cc:1468] Processing CreateTablet for tablet 46bb76c63ce749078365420f224d62f3 (DEFAULT_TABLE table=default.Table_825efeee6983439b9a45a82092f1e29e [id=ea2235e61e034f0c823898696bbec076]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:55.937927 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53660707e9204e2989a63b46d007140a consensus state has no leader
W20250629 01:51:55.938690 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76118a6e684448b09f244d44c909ccaa consensus state has no leader
I20250629 01:51:55.939079 881 tablet_service.cc:1468] Processing CreateTablet for tablet 999a62f27eb441699f641562ae60a287 (DEFAULT_TABLE table=default.Table_6e07ccb977954917826f8f245c6be17a [id=5a8e712d2cec42cf812be7aa2302e4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.940196 885 tablet_service.cc:1468] Processing CreateTablet for tablet 5552fced2b794ed5b219de28b2c307cf (DEFAULT_TABLE table=default.Table_d0e7545edb1e4ed1b67d3167c05b0995 [id=02bde06a6b2940a9be983b400783223c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:55.940567 712 consensus_peers.cc:489] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:55.940608 895 tablet_service.cc:1468] Processing CreateTablet for tablet 12d63924c5ff4a99ac62f89a9d29423b (DEFAULT_TABLE table=default.Table_dae51ebf34aa422a9e59b9369d5564a0 [id=f064321537ad49aeab2f63a840b16764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.940644 877 tablet_service.cc:1468] Processing CreateTablet for tablet df0992d1ba1b4aa39835e4db3622bcb8 (DEFAULT_TABLE table=default.Table_657a566bffb64d6293ed429ab1062bb7 [id=713ed3a3566345aba2869be0ef48ec81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.940937 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0992d1ba1b4aa39835e4db3622bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.940913 753 tablet_service.cc:1468] Processing CreateTablet for tablet df0992d1ba1b4aa39835e4db3622bcb8 (DEFAULT_TABLE table=default.Table_657a566bffb64d6293ed429ab1062bb7 [id=713ed3a3566345aba2869be0ef48ec81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.941129 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bb76c63ce749078365420f224d62f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.942301 748 tablet_service.cc:1468] Processing CreateTablet for tablet 9c66df5169224fea8c4fc45df7d695d8 (DEFAULT_TABLE table=default.Table_dee672b0788449d08349b1fecc061b43 [id=1cb14ee200584a3589957d109ce3004b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.943044 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 661f4bf68fc14cfdaa86ff86abc8ab74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.943465 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999a62f27eb441699f641562ae60a287. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.943884 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5552fced2b794ed5b219de28b2c307cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.944386 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:55.944936 896 tablet_service.cc:1468] Processing CreateTablet for tablet 9c66df5169224fea8c4fc45df7d695d8 (DEFAULT_TABLE table=default.Table_dee672b0788449d08349b1fecc061b43 [id=1cb14ee200584a3589957d109ce3004b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.945483 756 tablet_service.cc:1468] Processing CreateTablet for tablet fb03913296df45188e23337042e96d23 (DEFAULT_TABLE table=default.Table_f60d91fd7c1949c4ba988f3544ddb755 [id=1d42870d345b4f7eb93be692879b6f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.945693 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d63924c5ff4a99ac62f89a9d29423b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.948586 1032 tablet_service.cc:1468] Processing CreateTablet for tablet f7e6f33c7bee44d1a2b6feb0625faedf (DEFAULT_TABLE table=default.Table_3cd6904b29dc4d52bb6acb613babf904 [id=259db841e6904f4faddfafd0a0e63a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.950480 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7e6f33c7bee44d1a2b6feb0625faedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.951323 1155 tablet_bootstrap.cc:492] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:55.952631 1155 tablet_bootstrap.cc:654] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:55.954321 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999a62f27eb441699f641562ae60a287. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.956265 850 leader_election.cc:343] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:55.956357 850 leader_election.cc:304] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:55.956897 2217 raft_consensus.cc:2747] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:55.957005 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5c80397a2c4648d1b86ec4b46770a70b (DEFAULT_TABLE table=default.Table_8a512c677c164b1b861a7f53d5cb8b37 [id=835a4f7ca28e4d2695c8c33b7c5d4f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.957262 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f56d625aa5e456d84571b15184e1807. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.957262 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c80397a2c4648d1b86ec4b46770a70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.960352 878 tablet_service.cc:1468] Processing CreateTablet for tablet fb03913296df45188e23337042e96d23 (DEFAULT_TABLE table=default.Table_f60d91fd7c1949c4ba988f3544ddb755 [id=1d42870d345b4f7eb93be692879b6f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.960650 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb03913296df45188e23337042e96d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.963317 886 tablet_service.cc:1468] Processing CreateTablet for tablet 38252fbce581492fbb3db848a2a31a49 (DEFAULT_TABLE table=default.Table_0501e4a60a1046ad958d2d1d4e077bc9 [id=92e013d509e442d6b106cf6d1f76a5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.963567 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d60a3c1bc642969ced8c712e5af1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.967105 749 tablet_service.cc:1468] Processing CreateTablet for tablet b277782f84be461ca9247d15cda2f188 (DEFAULT_TABLE table=default.Table_8f48d7115c8b4e7a905249d06a3cb424 [id=af4b9fd6ecb24ed19031a702023ba345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.967355 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b277782f84be461ca9247d15cda2f188. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.968253 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a345f9896144379bc0d923ac517a0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.971755 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7806330425045ed809f092f20ada6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.972301 749 tablet_service.cc:1468] Processing CreateTablet for tablet 5c80397a2c4648d1b86ec4b46770a70b (DEFAULT_TABLE table=default.Table_8a512c677c164b1b861a7f53d5cb8b37 [id=835a4f7ca28e4d2695c8c33b7c5d4f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.972558 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c80397a2c4648d1b86ec4b46770a70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.979296 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8be7d65065fa4ed6956cad11d2cb5ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.982806 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 77727efeef5e4f0c89f4afca2941d02a (DEFAULT_TABLE table=default.Table_45ad1da33f23408c858cd346e7ac71bf [id=b59ee7ba9aba4cd1b61e32be0ab1dce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.983114 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77727efeef5e4f0c89f4afca2941d02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.984057 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2818fd021c504a3aa687ad16bf8a0c48. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:55.987416 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 7908afa2abaa4521b44f954e0790f986 (table default.Table_e998111141ca474dbff482e945a67465 [id=a3739e0c0f154fec8a5ffbf6b2b08fcc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7908afa2abaa4521b44f954e0790f986 consensus state has no leader
I20250629 01:51:55.988257 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e50eb762144fc28c531af7514776b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.989071 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet add482b533d045e7a3010064fae730cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.989114 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.989269 1158 tablet_replica.cc:331] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:55.989424 1158 raft_consensus.cc:2241] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.989567 1158 raft_consensus.cc:2270] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.989840 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.989982 1861 raft_consensus.cc:491] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:55.990088 1861 raft_consensus.cc:513] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:55.990336 1158 ts_tablet_manager.cc:1905] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:55.990622 1861 leader_election.cc:290] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:55.991923 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 556c2050d2cb40f28a3a0bfb9d677d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.992264 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c2c00c5f7b4cefb88523b8f5453c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.992903 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631ca7239f574441b37c11e104c3d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.993270 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet d54bdcf7e1a14c669472af8acf36aac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:55.993516 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1ba1fec9bf49a5a44a193802edde0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.994220 1022 tablet_service.cc:1468] Processing CreateTablet for tablet b1ca655a041b4aff95df14041a51b6cf (DEFAULT_TABLE table=default.Table_e4309edc8dd64fc88947c246a80faac7 [id=2df5022bc3884ebcb91ba7353c5b3bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.996420 1158 ts_tablet_manager.cc:1918] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:55.996515 1158 log.cc:1199] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/add482b533d045e7a3010064fae730cf
I20250629 01:51:55.996805 1158 ts_tablet_manager.cc:1939] T add482b533d045e7a3010064fae730cf P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:55.968677 890 tablet_service.cc:1468] Processing CreateTablet for tablet d2870ea06d364d2cb3a00d428b831df3 (DEFAULT_TABLE table=default.Table_254111a6f5a74f2c9f3b57c27da3b8a3 [id=979ea777ca4349cdbf0291c16b1723f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:55.976735 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c66df5169224fea8c4fc45df7d695d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.997290 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38252fbce581492fbb3db848a2a31a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:55.997494 1158 tablet_replica.cc:331] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:55.997632 1158 raft_consensus.cc:2241] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:55.997726 1158 raft_consensus.cc:2270] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:55.998091 1158 ts_tablet_manager.cc:1905] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.000447 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2870ea06d364d2cb3a00d428b831df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.002969 1158 ts_tablet_manager.cc:1918] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.003063 1158 log.cc:1199] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d6bf3c7d0bb548e0b77ba8b569a72d0d
I20250629 01:51:56.003297 1158 ts_tablet_manager.cc:1939] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:55.980141 884 tablet_service.cc:1468] Processing CreateTablet for tablet b277782f84be461ca9247d15cda2f188 (DEFAULT_TABLE table=default.Table_8f48d7115c8b4e7a905249d06a3cb424 [id=af4b9fd6ecb24ed19031a702023ba345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.003641 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b277782f84be461ca9247d15cda2f188. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.006606 2273 raft_consensus.cc:491] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.006693 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 18df997317c04431a496448a4836ac2a (DEFAULT_TABLE table=default.Table_4356a70d10bd41e4ad6b77e5612a81ef [id=50b83fb9661b4e0f9ea0e5003ba63b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.006709 2273 raft_consensus.cc:513] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.007014 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18df997317c04431a496448a4836ac2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.007287 2273 leader_election.cc:290] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.007699 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.008138 988 leader_election.cc:343] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.008426 885 tablet_service.cc:1468] Processing CreateTablet for tablet a4c5067d924946fabd0b28ab5a68d2ee (DEFAULT_TABLE table=default.Table_f5624ebd8475423285371a2a55453a30 [id=d61c01c2f21f4b0686ba59324fb383f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.008679 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c5067d924946fabd0b28ab5a68d2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.008687 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.010326 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 60c79b5f0c1a4a9b82c9f57f43cf2a64 (DEFAULT_TABLE table=default.Table_02190f4ee1a746e7b458b005042fbc0f [id=b5367f695a8645a4896ab11f648bcec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.010413 1023 tablet_service.cc:1468] Processing CreateTablet for tablet feeaa50e7f224da69128ea916e4145fd (DEFAULT_TABLE table=default.Table_e646217c593b412f8d30f34145ea5325 [id=2b6021fe8aef44e7a806e56489e34c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.010524 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 1a05cd13ca124ff399e10b4ca79b6ccf (DEFAULT_TABLE table=default.Table_5257d4d9f670497287003f5c8bf98456 [id=4266d70c7a514dc8af01519db5002340]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.010607 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60c79b5f0c1a4a9b82c9f57f43cf2a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.010764 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a05cd13ca124ff399e10b4ca79b6ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.010938 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feeaa50e7f224da69128ea916e4145fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.010906 2217 raft_consensus.cc:491] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.011005 2217 raft_consensus.cc:513] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.011458 2217 leader_election.cc:290] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:56.010437 1017 tablet_service.cc:1468] Processing CreateTablet for tablet ba9b705b1f8c4292ab33bc47aab3ca12 (DEFAULT_TABLE table=default.Table_85e8350afe3f4983b2225325b3c58208 [id=837588cbd00f48caafa3c004e2bd852b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.011713 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba9b705b1f8c4292ab33bc47aab3ca12. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:56.017542 712 consensus_peers.cc:489] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:55.985713 1157 ts_tablet_manager.cc:1918] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:55.993803 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ba9eabd06174b87a1485f4921895c5f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:56.017781 1157 log.cc:1199] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/683c87ea2ad44cecb2366d4a846d9195
I20250629 01:51:56.018055 1157 ts_tablet_manager.cc:1939] T 683c87ea2ad44cecb2366d4a846d9195 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:51:56.018033 712 leader_election.cc:343] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.018141 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 23d539ff02dd4f2ca5a8f070276bf901 (DEFAULT_TABLE table=default.Table_c3c28291b8e046aebf0a8accc24e0599 [id=e7265d794ecc467596c30273910dd873]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.018405 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23d539ff02dd4f2ca5a8f070276bf901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.019052 1157 tablet_replica.cc:331] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.019199 1157 raft_consensus.cc:2241] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.019305 1157 raft_consensus.cc:2270] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:56.019709 850 leader_election.cc:343] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8799b1858bf2408a85f0956578639393
I20250629 01:51:56.020143 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 683c87ea2ad44cecb2366d4a846d9195 (table default.Table_04cc9d29a25645f0b3facf31bd7a9f57 [id=395f2b984ff9443a90e78e8d44bca991]) successfully deleted
W20250629 01:51:56.020357 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ee1baa976b5e47e490a0fd561b37f83d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1baa976b5e47e490a0fd561b37f83d already in progress: creating tablet
I20250629 01:51:56.020375 754 tablet_service.cc:1468] Processing CreateTablet for tablet a1a88bf7668546f4b303dcddba10d1a0 (DEFAULT_TABLE table=default.Table_80d068bf778042d1b60dfbad2b5e36a7 [id=cd3d373467b54508a8edfc36da09c575]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:56.020541 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet caa63fe10ff9487a9fc16479285c5abd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caa63fe10ff9487a9fc16479285c5abd already in progress: creating tablet
I20250629 01:51:56.020507 878 tablet_service.cc:1468] Processing CreateTablet for tablet a1a88bf7668546f4b303dcddba10d1a0 (DEFAULT_TABLE table=default.Table_80d068bf778042d1b60dfbad2b5e36a7 [id=cd3d373467b54508a8edfc36da09c575]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:56.020716 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8cea647451954d8ab3bd495a3a080c97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cea647451954d8ab3bd495a3a080c97 already in progress: creating tablet
I20250629 01:51:56.020943 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a88bf7668546f4b303dcddba10d1a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:56.020921 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 06d591c66d734f76a1bda2194ec62776 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06d591c66d734f76a1bda2194ec62776 already in progress: creating tablet
I20250629 01:51:56.022732 896 tablet_service.cc:1468] Processing CreateTablet for tablet 72a9ca562aa8445e9b26b789d1ebc02b (DEFAULT_TABLE table=default.Table_7e5e094e14304fd7b0899ba75536ea4a [id=d1057a419e14492295a2c90460a822d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.023027 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a9ca562aa8445e9b26b789d1ebc02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.023747 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 89aa7f502b1a4198aea2096105dd00d6 (DEFAULT_TABLE table=default.Table_3c3ba33385624970b8f09248b76e5cc1 [id=d34b612ad43d4067869c532a99f86d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.023806 877 tablet_service.cc:1468] Processing CreateTablet for tablet a1ddbbbe43044a70a80e7dd66011de81 (DEFAULT_TABLE table=default.Table_8a9b6060d8cd47aa8e1019d961b2fbab [id=e0992897320747a9852e47acdbc46b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.024020 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89aa7f502b1a4198aea2096105dd00d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.024191 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ddbbbe43044a70a80e7dd66011de81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.025218 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0b847d6ea2594d1880f4d7a9d7e825e8 (DEFAULT_TABLE table=default.Table_11fa9f38439b4f6d990266584eebfaf9 [id=76187ce4b2f349fbb833b92b6ac6f642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.025959 1159 tablet_bootstrap.cc:492] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.026108 1159 ts_tablet_manager.cc:1397] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.268s user 0.012s sys 0.026s
I20250629 01:51:56.026710 1159 raft_consensus.cc:357] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.026917 1159 raft_consensus.cc:383] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.026997 1159 raft_consensus.cc:738] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.027206 1159 consensus_queue.cc:260] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.027514 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d63924c5ff4a99ac62f89a9d29423b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.027509 740 tablet_service.cc:1468] Processing CreateTablet for tablet 38252fbce581492fbb3db848a2a31a49 (DEFAULT_TABLE table=default.Table_0501e4a60a1046ad958d2d1d4e077bc9 [id=92e013d509e442d6b106cf6d1f76a5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.028470 755 tablet_service.cc:1468] Processing CreateTablet for tablet a4c5067d924946fabd0b28ab5a68d2ee (DEFAULT_TABLE table=default.Table_f5624ebd8475423285371a2a55453a30 [id=d61c01c2f21f4b0686ba59324fb383f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.028716 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:56.028884 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bed6b19b8454766add0f458a1eac7c4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:56.029327 986 leader_election.cc:343] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.029404 986 leader_election.cc:304] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.029685 2273 raft_consensus.cc:2747] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:56.029870 850 leader_election.cc:343] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 8799b1858bf2408a85f0956578639393
I20250629 01:51:56.029963 850 leader_election.cc:304] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.030230 2217 raft_consensus.cc:2747] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.030598 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5552fced2b794ed5b219de28b2c307cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.031724 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0818e53ec2ef4adeba03cabe3d5dfcc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:56.031975 1155 tablet_bootstrap.cc:492] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.032078 1155 ts_tablet_manager.cc:1397] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.081s user 0.000s sys 0.010s
W20250629 01:51:56.032258 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0818e53ec2ef4adeba03cabe3d5dfcc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.033690 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:56.037321 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18a5af7b1c8b40dd8751d5d7a694c3eb consensus state has no leader
I20250629 01:51:56.037364 890 tablet_service.cc:1468] Processing CreateTablet for tablet e4e34e745ad04ab986da7d22d307ba69 (DEFAULT_TABLE table=default.Table_5377f24c99e3434a805be34e0bcb9a39 [id=79545f981abb49d3aa9e1ac960f9a443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.019719 1157 ts_tablet_manager.cc:1905] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:56.037719 712 consensus_peers.cc:489] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.038378 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bb76c63ce749078365420f224d62f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.038957 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_78b4b6aea1c144eeb3669ca5d081e0db" } modify_external_catalogs: true
I20250629 01:51:56.039113 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_78b4b6aea1c144eeb3669ca5d081e0db" } modify_external_catalogs: true
I20250629 01:51:56.040324 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2870ea06d364d2cb3a00d428b831df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.040513 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_97c5167bc43d47699812760d5d1939c9" } modify_external_catalogs: true
I20250629 01:51:56.040656 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_97c5167bc43d47699812760d5d1939c9" } modify_external_catalogs: true
I20250629 01:51:56.041718 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_54c1b81ff73b436fbf764565c0ab55ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.042167 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54c1b81ff73b436fbf764565c0ab55ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:56.043190 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d932c3240624c07ab5c43f70112f326 consensus state has no leader
I20250629 01:51:56.043454 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b2805bb726134583891dea5763bcb7c1" } modify_external_catalogs: true
I20250629 01:51:56.043622 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b2805bb726134583891dea5763bcb7c1" } modify_external_catalogs: true
I20250629 01:51:56.045231 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_583444e245fc499bb4f38e7921e59da3" } modify_external_catalogs: true
I20250629 01:51:56.045352 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_583444e245fc499bb4f38e7921e59da3" } modify_external_catalogs: true
I20250629 01:51:56.046797 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 8dd084685cc0407da9e12b0a4df386e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.048695 881 tablet_service.cc:1468] Processing CreateTablet for tablet e2a9eb99f0a748818fa44cd9f69a6d70 (DEFAULT_TABLE table=default.Table_cd351e2fabea4665b7954d5eca538f02 [id=a49b39d0f40b468a8f96e07e3668c15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.055858 1159 ts_tablet_manager.cc:1428] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.030s user 0.000s sys 0.001s
I20250629 01:51:56.057165 2273 raft_consensus.cc:491] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.057277 2273 raft_consensus.cc:513] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.057773 2273 leader_election.cc:290] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.058122 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f38ccc485043ba8b8a6f644a3e4f36" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.058460 988 leader_election.cc:343] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.058799 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f38ccc485043ba8b8a6f644a3e4f36" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:56.060047 986 leader_election.cc:343] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.060159 986 leader_election.cc:304] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.060497 2273 raft_consensus.cc:2747] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.067773 1155 raft_consensus.cc:357] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.067914 1155 raft_consensus.cc:383] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.067963 1155 raft_consensus.cc:738] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.068149 1155 consensus_queue.cc:260] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.068522 1155 ts_tablet_manager.cc:1428] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.036s user 0.001s sys 0.000s
I20250629 01:51:56.070642 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 661f4bf68fc14cfdaa86ff86abc8ab74 (DEFAULT_TABLE table=default.Table_3efca07702364cf982f01059b4d3899c [id=3aa699feb56c4af5b4052779edd43bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.070938 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 661f4bf68fc14cfdaa86ff86abc8ab74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.074358 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0992d1ba1b4aa39835e4db3622bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.077874 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 4a345f9896144379bc0d923ac517a0ed (DEFAULT_TABLE table=default.Table_716fe511fb2c4fb384a34d9964136cab [id=4492edac31974b02900902435520113b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.077956 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 2818fd021c504a3aa687ad16bf8a0c48 (DEFAULT_TABLE table=default.Table_d5042d6281a54e70861a5619a2967981 [id=e050eacfc013416fad52f49422828078]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.078002 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 8be7d65065fa4ed6956cad11d2cb5ffb (DEFAULT_TABLE table=default.Table_29b731f4ff984ce5bb98b903fbb804c0 [id=26d79160a7d94ba1a1dd4c0e25d52278]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.078147 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a345f9896144379bc0d923ac517a0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.078320 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8be7d65065fa4ed6956cad11d2cb5ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.078408 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 8f56d625aa5e456d84571b15184e1807 (DEFAULT_TABLE table=default.Table_b296280dfbe1428b9e98ebbb01dacc44 [id=cf990cf769874bd88531d52c02f22a1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.078678 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f56d625aa5e456d84571b15184e1807. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.079715 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c66df5169224fea8c4fc45df7d695d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.080179 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2818fd021c504a3aa687ad16bf8a0c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.084283 1159 tablet_bootstrap.cc:492] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.084592 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb03913296df45188e23337042e96d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.085084 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a88bf7668546f4b303dcddba10d1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.088442 750 tablet_service.cc:1468] Processing CreateTablet for tablet a1ddbbbe43044a70a80e7dd66011de81 (DEFAULT_TABLE table=default.Table_8a9b6060d8cd47aa8e1019d961b2fbab [id=e0992897320747a9852e47acdbc46b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.088727 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ddbbbe43044a70a80e7dd66011de81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.089215 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c7806330425045ed809f092f20ada6fd (DEFAULT_TABLE table=default.Table_551758fbd6ed485096040f978cfec8af [id=3594a358e67e46edbff8113ca4ea7e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.089459 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7806330425045ed809f092f20ada6fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.092403 745 tablet_service.cc:1468] Processing CreateTablet for tablet 72a9ca562aa8445e9b26b789d1ebc02b (DEFAULT_TABLE table=default.Table_7e5e094e14304fd7b0899ba75536ea4a [id=d1057a419e14492295a2c90460a822d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.092805 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a9ca562aa8445e9b26b789d1ebc02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.093181 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0b847d6ea2594d1880f4d7a9d7e825e8 (DEFAULT_TABLE table=default.Table_11fa9f38439b4f6d990266584eebfaf9 [id=76187ce4b2f349fbb833b92b6ac6f642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.093516 2273 raft_consensus.cc:491] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.093551 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b847d6ea2594d1880f4d7a9d7e825e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.093616 2273 raft_consensus.cc:513] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.094139 2273 leader_election.cc:290] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.094689 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:56.094691 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.095096 988 leader_election.cc:343] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:56.095309 986 leader_election.cc:343] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.095382 986 leader_election.cc:304] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.095718 2273 raft_consensus.cc:2747] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.096249 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38252fbce581492fbb3db848a2a31a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.096347 756 tablet_service.cc:1468] Processing CreateTablet for tablet e4e34e745ad04ab986da7d22d307ba69 (DEFAULT_TABLE table=default.Table_5377f24c99e3434a805be34e0bcb9a39 [id=79545f981abb49d3aa9e1ac960f9a443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.096578 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4e34e745ad04ab986da7d22d307ba69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.100415 741 tablet_service.cc:1468] Processing CreateTablet for tablet e2a9eb99f0a748818fa44cd9f69a6d70 (DEFAULT_TABLE table=default.Table_cd351e2fabea4665b7954d5eca538f02 [id=a49b39d0f40b468a8f96e07e3668c15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.100661 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a9eb99f0a748818fa44cd9f69a6d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.102773 748 tablet_service.cc:1468] Processing CreateTablet for tablet 9f6fbefddf434f32ab373b4e0191cee2 (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.103020 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6fbefddf434f32ab373b4e0191cee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.103612 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ba9eabd06174b87a1485f4921895c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.103749 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c5067d924946fabd0b28ab5a68d2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.026114 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b847d6ea2594d1880f4d7a9d7e825e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.103612 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 with delete_type TABLET_DATA_DELETED (replaced by tablet d73c2eb0d69e467b94a250e1790d8142) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.108381 1159 tablet_bootstrap.cc:654] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.110939 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4e34e745ad04ab986da7d22d307ba69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.113749 1155 tablet_bootstrap.cc:492] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.113987 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7aef2514a87c4c4aaf4d51b314630185 (DEFAULT_TABLE table=default.Table_a1e8b8763bff46768cb4e15ba61d470b [id=9d8c7b07069d4b86b2a5e18fa3e88ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.114244 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aef2514a87c4c4aaf4d51b314630185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.114802 1155 tablet_bootstrap.cc:654] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.119065 1157 ts_tablet_manager.cc:1918] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.119223 1157 log.cc:1199] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/75ce1bd4af434f7e99092aafc0ab0118
I20250629 01:51:56.119654 1157 ts_tablet_manager.cc:1939] T 75ce1bd4af434f7e99092aafc0ab0118 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.120638 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a9eb99f0a748818fa44cd9f69a6d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.121026 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 75ce1bd4af434f7e99092aafc0ab0118 (table default.Table_d9bb8cc1cab544148faa53db857172d3 [id=5d8dd5f1acfd4c57b7080fc663a1d32d]) successfully deleted
I20250629 01:51:56.121363 1157 tablet_replica.cc:331] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.121659 1157 raft_consensus.cc:2241] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.121902 1157 raft_consensus.cc:2270] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.122625 1157 ts_tablet_manager.cc:1905] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.124372 1158 tablet_replica.cc:331] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:56.124536 1158 raft_consensus.cc:2241] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.124665 1158 raft_consensus.cc:2270] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.125304 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_79b81d65f78949388e3fbee231ed031c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.125756 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79b81d65f78949388e3fbee231ed031c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.125875 1158 ts_tablet_manager.cc:1905] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.130472 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3fce5fefd74344c29e1895554190fd44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:56.131716 1156 ts_tablet_manager.cc:1918] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.131845 1156 log.cc:1199] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9bbbeb43473a4f6f892056a30f62589f
I20250629 01:51:56.132158 1156 ts_tablet_manager.cc:1939] T 9bbbeb43473a4f6f892056a30f62589f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:56.132213 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fce5fefd74344c29e1895554190fd44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.131780 450 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:56.133531 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5d856d8a80084e8bae95e734dff72636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:56.133667 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ad1ba1fec9bf49a5a44a193802edde0a (DEFAULT_TABLE table=default.Table_efd88358bbde43528c5d3013bb7aaaf2 [id=05c1626a711d405996ae1c40256e4414]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:56.133868 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d856d8a80084e8bae95e734dff72636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.133944 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad1ba1fec9bf49a5a44a193802edde0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.136931 1159 tablet_bootstrap.cc:492] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.137033 1159 ts_tablet_manager.cc:1397] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.053s user 0.000s sys 0.004s
I20250629 01:51:56.137095 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0501e4a60a1046ad958d2d1d4e077bc9" } new_table_name: "default.Table_9ef21c0a6bf74bd49725f221cb0b5b66" modify_external_catalogs: true
I20250629 01:51:56.137594 1159 raft_consensus.cc:357] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.138818 1159 raft_consensus.cc:383] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.138875 1159 raft_consensus.cc:738] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.139040 1159 consensus_queue.cc:260] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.139652 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ce90c045115844b0b96143f91f1be1d4
I20250629 01:51:56.140372 877 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.141069 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8873b53d7e42492da64f7f677e05c6f1
I20250629 01:51:56.141638 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:56.142789 744 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.142985 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 8873b53d7e42492da64f7f677e05c6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.143028 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 631ca7239f574441b37c11e104c3d56b (DEFAULT_TABLE table=default.Table_39a0ff1dd8064dc4a2990b95fac35099 [id=811dcd76b5c742f69d611cce6f6bde86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.143354 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631ca7239f574441b37c11e104c3d56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.144632 1159 ts_tablet_manager.cc:1428] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250629 01:51:56.145856 1159 tablet_bootstrap.cc:492] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.147220 1159 tablet_bootstrap.cc:654] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.141964 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fc9218fdbd474925bd2444d014e3ba06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.147897 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc9218fdbd474925bd2444d014e3ba06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.148831 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 8873b53d7e42492da64f7f677e05c6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.149122 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2e9e6727c5e3422ab1463298cf3ec974
I20250629 01:51:56.149650 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 2e9e6727c5e3422ab1463298cf3ec974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.150218 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_151fdf3b919844009a4deed7768b613b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.150672 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_151fdf3b919844009a4deed7768b613b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.150799 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 2e9e6727c5e3422ab1463298cf3ec974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.151197 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 12d63924c5ff4a99ac62f89a9d29423b (DEFAULT_TABLE table=default.Table_dae51ebf34aa422a9e59b9369d5564a0 [id=f064321537ad49aeab2f63a840b16764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.151845 1015 tablet_service.cc:1468] Processing CreateTablet for tablet b6d60a3c1bc642969ced8c712e5af1c1 (DEFAULT_TABLE table=default.Table_20c6268f43c64a5480574ad87d428a7c [id=1b897d7a629e4a26998e86e4f1ebb33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.154258 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b7e50eb762144fc28c531af7514776b0 (DEFAULT_TABLE table=default.Table_6f23d8816c084fafa5c57899912ffdd6 [id=c52afe871a3d49e980d364ac0e39d14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.154836 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 8bed6b19b8454766add0f458a1eac7c4 (DEFAULT_TABLE table=default.Table_adcbb5e4f5cb4ae78e1f48f042c4033e [id=3391dd3dacc2450fbb71e469e866ad78]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.155601 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12d63924c5ff4a99ac62f89a9d29423b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.155813 1025 tablet_service.cc:1468] Processing CreateTablet for tablet d2870ea06d364d2cb3a00d428b831df3 (DEFAULT_TABLE table=default.Table_254111a6f5a74f2c9f3b57c27da3b8a3 [id=979ea777ca4349cdbf0291c16b1723f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.156136 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2870ea06d364d2cb3a00d428b831df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.160203 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bed6b19b8454766add0f458a1eac7c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.161902 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:51:56.163533 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cca695fad03947debe4d1d515b3c73d8" } new_table_name: "default.Table_7fff96135543443f861dc133aea7b971" modify_external_catalogs: true
I20250629 01:51:56.164588 886 tablet_service.cc:1515] Processing DeleteTablet for tablet f8b38c20eb1849b1a9f8acd40315eef5 with delete_type TABLET_DATA_DELETED (replaced by tablet c0b9afb9976541b2bc7ab188f35a62ec) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.166308 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d60a3c1bc642969ced8c712e5af1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.167430 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0f650c0d6e344c3883bd177ecd2a699f" } modify_external_catalogs: true
I20250629 01:51:56.167557 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0f650c0d6e344c3883bd177ecd2a699f" } modify_external_catalogs: true
I20250629 01:51:56.170976 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1004) took 1797 ms. Trace:
I20250629 01:51:56.171096 1156 rpcz_store.cc:276] 0629 01:51:54.373601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.373654 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:55.739412 (+1365758us) ts_tablet_manager.cc:1145] Deleting tablet 9bbbeb43473a4f6f892056a30f62589f
0629 01:51:55.739421 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:55.754023 (+ 14602us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:56.131636 (+377613us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:56.170960 (+ 39324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365154}
I20250629 01:51:56.171562 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9bbbeb43473a4f6f892056a30f62589f (table default.Table_dc8c2079eef0455a8ba102bd39414e15 [id=8705c82c65df415abf3299b10b90f48c]) successfully deleted
I20250629 01:51:56.172971 1159 tablet_bootstrap.cc:492] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.173067 1159 ts_tablet_manager.cc:1397] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.002s
I20250629 01:51:56.173676 1159 raft_consensus.cc:357] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.173802 1159 raft_consensus.cc:383] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.173841 1159 raft_consensus.cc:738] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.174010 1159 consensus_queue.cc:260] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.176141 2217 raft_consensus.cc:491] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.176245 2217 raft_consensus.cc:513] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.176712 2217 leader_election.cc:290] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:56.179136 850 leader_election.cc:343] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: b0c2dd05f5bb466fbe38ed259ce3a4b8
W20250629 01:51:56.179647 850 leader_election.cc:343] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: b0c2dd05f5bb466fbe38ed259ce3a4b8
I20250629 01:51:56.179735 850 leader_election.cc:304] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.180119 2217 raft_consensus.cc:2747] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.180200 1159 ts_tablet_manager.cc:1428] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250629 01:51:56.181464 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 9c66df5169224fea8c4fc45df7d695d8 (DEFAULT_TABLE table=default.Table_dee672b0788449d08349b1fecc061b43 [id=1cb14ee200584a3589957d109ce3004b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.181735 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c66df5169224fea8c4fc45df7d695d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.181787 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 38252fbce581492fbb3db848a2a31a49 (DEFAULT_TABLE table=default.Table_0501e4a60a1046ad958d2d1d4e077bc9 [id=92e013d509e442d6b106cf6d1f76a5e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.182051 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38252fbce581492fbb3db848a2a31a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.184276 1013 tablet_service.cc:1468] Processing CreateTablet for tablet fb03913296df45188e23337042e96d23 (DEFAULT_TABLE table=default.Table_f60d91fd7c1949c4ba988f3544ddb755 [id=1d42870d345b4f7eb93be692879b6f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.184520 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb03913296df45188e23337042e96d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.184437 1157 ts_tablet_manager.cc:1918] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.184688 1157 log.cc:1199] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/196f1bd233ab4399a9d22a55b8bc2635
I20250629 01:51:56.185369 1021 tablet_service.cc:1468] Processing CreateTablet for tablet df0992d1ba1b4aa39835e4db3622bcb8 (DEFAULT_TABLE table=default.Table_657a566bffb64d6293ed429ab1062bb7 [id=713ed3a3566345aba2869be0ef48ec81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.185439 1157 ts_tablet_manager.cc:1939] T 196f1bd233ab4399a9d22a55b8bc2635 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.185604 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0992d1ba1b4aa39835e4db3622bcb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.187225 1020 tablet_service.cc:1468] Processing CreateTablet for tablet a1a88bf7668546f4b303dcddba10d1a0 (DEFAULT_TABLE table=default.Table_80d068bf778042d1b60dfbad2b5e36a7 [id=cd3d373467b54508a8edfc36da09c575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.187376 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 196f1bd233ab4399a9d22a55b8bc2635 (table default.Table_821e341e379740a59615270511038004 [id=910c490dcc294f58b93e5b6c4846d263]) successfully deleted
I20250629 01:51:56.187466 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1a88bf7668546f4b303dcddba10d1a0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:56.188273 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2521a476643c41c1a2e5282755a58091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2521a476643c41c1a2e5282755a58091 already in progress: creating tablet
I20250629 01:51:56.188279 1027 tablet_service.cc:1468] Processing CreateTablet for tablet b277782f84be461ca9247d15cda2f188 (DEFAULT_TABLE table=default.Table_8f48d7115c8b4e7a905249d06a3cb424 [id=af4b9fd6ecb24ed19031a702023ba345]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.188560 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b277782f84be461ca9247d15cda2f188. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:56.189038 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 3ba5887dffc1460a89208447ecc6a198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ba5887dffc1460a89208447ecc6a198 already in progress: creating tablet
W20250629 01:51:56.189800 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a2e9de48e4ad401880f351f2bd55cf5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e9de48e4ad401880f351f2bd55cf5c already in progress: creating tablet
W20250629 01:51:56.190495 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 303f77a729c648f9b1146d11056b242d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303f77a729c648f9b1146d11056b242d already in progress: creating tablet
I20250629 01:51:56.190944 1157 tablet_replica.cc:331] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.191339 1157 raft_consensus.cc:2241] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.191699 1157 raft_consensus.cc:2270] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.192269 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 5c80397a2c4648d1b86ec4b46770a70b (DEFAULT_TABLE table=default.Table_8a512c677c164b1b861a7f53d5cb8b37 [id=835a4f7ca28e4d2695c8c33b7c5d4f09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.192528 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c80397a2c4648d1b86ec4b46770a70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.192479 1016 tablet_service.cc:1468] Processing CreateTablet for tablet a1ddbbbe43044a70a80e7dd66011de81 (DEFAULT_TABLE table=default.Table_8a9b6060d8cd47aa8e1019d961b2fbab [id=e0992897320747a9852e47acdbc46b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.192780 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1ddbbbe43044a70a80e7dd66011de81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.192917 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c951e1bffffa4ed7b7fcff1b87d9ee23" } new_table_name: "default.Table_4a672330f8e74cb185fab662d8365cfe" modify_external_catalogs: true
I20250629 01:51:56.193079 1157 ts_tablet_manager.cc:1905] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.195241 877 tablet_service.cc:1468] Processing CreateTablet for tablet 9f6fbefddf434f32ab373b4e0191cee2 (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.195501 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6fbefddf434f32ab373b4e0191cee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.197896 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 72a9ca562aa8445e9b26b789d1ebc02b (DEFAULT_TABLE table=default.Table_7e5e094e14304fd7b0899ba75536ea4a [id=d1057a419e14492295a2c90460a822d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.198133 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a9ca562aa8445e9b26b789d1ebc02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.199954 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9f6fbefddf434f32ab373b4e0191cee2
I20250629 01:51:56.200527 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1ac60a024a074953b513ae4f60840d25 (DEFAULT_TABLE table=default.Table_b0da5a36b5c141cc848f23ee43a30fa1 [id=1b2e5bc17a524432af71014b105fd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.200850 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ac60a024a074953b513ae4f60840d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.204707 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 46bb76c63ce749078365420f224d62f3 (DEFAULT_TABLE table=default.Table_825efeee6983439b9a45a82092f1e29e [id=ea2235e61e034f0c823898696bbec076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.205018 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46bb76c63ce749078365420f224d62f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.205200 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ea681bd91e54e4898bcfb72bed6198f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.208194 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7e50eb762144fc28c531af7514776b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.210343 1159 tablet_bootstrap.cc:492] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.212324 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 999a62f27eb441699f641562ae60a287 (DEFAULT_TABLE table=default.Table_6e07ccb977954917826f8f245c6be17a [id=5a8e712d2cec42cf812be7aa2302e4a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.212630 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 999a62f27eb441699f641562ae60a287. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.213797 1159 tablet_bootstrap.cc:654] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.213878 751 tablet_service.cc:1468] Processing CreateTablet for tablet 1ac60a024a074953b513ae4f60840d25 (DEFAULT_TABLE table=default.Table_b0da5a36b5c141cc848f23ee43a30fa1 [id=1b2e5bc17a524432af71014b105fd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.214248 913 raft_consensus.cc:3058] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.214460 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ac60a024a074953b513ae4f60840d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.217123 1154 tablet_bootstrap.cc:492] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.217253 1154 ts_tablet_manager.cc:1397] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.288s user 0.003s sys 0.005s
I20250629 01:51:56.217983 1154 raft_consensus.cc:357] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.218147 1154 raft_consensus.cc:383] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.218225 1154 raft_consensus.cc:738] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.218403 1154 consensus_queue.cc:260] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.220201 1861 raft_consensus.cc:491] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.220304 1861 raft_consensus.cc:513] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.220746 1861 leader_election.cc:290] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.221535 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.221890 712 leader_election.cc:343] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.222121 1154 ts_tablet_manager.cc:1428] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:51:56.222343 1154 tablet_bootstrap.cc:492] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.223425 1154 tablet_bootstrap.cc:654] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.224195 1156 tablet_replica.cc:331] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:56.224381 1156 raft_consensus.cc:2241] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.224537 1156 raft_consensus.cc:2270] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.225109 1156 ts_tablet_manager.cc:1905] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.230374 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f6fbefddf434f32ab373b4e0191cee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.230867 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 5552fced2b794ed5b219de28b2c307cf (DEFAULT_TABLE table=default.Table_d0e7545edb1e4ed1b67d3167c05b0995 [id=02bde06a6b2940a9be983b400783223c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.231168 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5552fced2b794ed5b219de28b2c307cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:56.232571 712 consensus_peers.cc:489] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 798516e55ca747fcb2942026adb78af0. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.233716 1155 tablet_bootstrap.cc:492] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.234408 1155 ts_tablet_manager.cc:1397] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.007s
I20250629 01:51:56.235124 1155 raft_consensus.cc:357] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.235738 1157 ts_tablet_manager.cc:1918] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.235886 1157 log.cc:1199] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e83009c1599a43d18d1c73c0101ab0be
I20250629 01:51:56.236346 1157 ts_tablet_manager.cc:1939] T e83009c1599a43d18d1c73c0101ab0be P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.238579 2328 tablet.cc:1722] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:56.239535 1155 raft_consensus.cc:383] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.241804 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]). Expected version 1 got 0
I20250629 01:51:56.243983 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e83009c1599a43d18d1c73c0101ab0be (table default.Table_6b7f597bfe3947f884cf5e129049c99b [id=e4518081a6904de8a019b6d2d015d7b7]) successfully deleted
I20250629 01:51:56.245927 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_648891736c08404eb59b4792a5970064" } new_table_name: "default.Table_3dcc0af85f1241a1a768b66f237b4087" modify_external_catalogs: true
I20250629 01:51:56.245937 446 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]). Expected version 1 got 0
W20250629 01:51:56.248809 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9ed1ee73bb8f4761806a66f988b57bf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed1ee73bb8f4761806a66f988b57bf9 already in progress: creating tablet
I20250629 01:51:56.251224 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_db4b4ae9a38f4d75845beeffd7251609" } new_table_name: "default.Table_1e35227a427f40c4b30cf8d97cbfc37f" modify_external_catalogs: true
I20250629 01:51:56.248515 1861 raft_consensus.cc:491] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.253257 1861 raft_consensus.cc:513] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.253748 1861 leader_election.cc:290] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.256275 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e4e34e745ad04ab986da7d22d307ba69 (DEFAULT_TABLE table=default.Table_5377f24c99e3434a805be34e0bcb9a39 [id=79545f981abb49d3aa9e1ac960f9a443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.256554 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 with delete_type TABLET_DATA_DELETED (replaced by tablet d73c2eb0d69e467b94a250e1790d8142) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.256556 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4e34e745ad04ab986da7d22d307ba69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.256891 1013 tablet_service.cc:1468] Processing CreateTablet for tablet e2a9eb99f0a748818fa44cd9f69a6d70 (DEFAULT_TABLE table=default.Table_cd351e2fabea4665b7954d5eca538f02 [id=a49b39d0f40b468a8f96e07e3668c15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.257136 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a9eb99f0a748818fa44cd9f69a6d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.258730 1155 raft_consensus.cc:738] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.259243 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_03d275243cf84467ab2396313192c17c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.259645 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03d275243cf84467ab2396313192c17c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.260178 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet 2e9e6727c5e3422ab1463298cf3ec974 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.260638 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 8873b53d7e42492da64f7f677e05c6f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.239416 881 tablet_service.cc:1468] Processing CreateTablet for tablet 56b13ca86979406e8d8a56f6e89201f8 (DEFAULT_TABLE table=default.Table_4e7128f0007c44f691cd8bc0931dfe17 [id=76f64c090c6e4d2abd1bc152f42b9011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.248752 1157 tablet_replica.cc:331] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.254331 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56b60431b72e4792aa1c32fdee40cdb9" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:56.261773 1157 raft_consensus.cc:2241] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.261888 1157 raft_consensus.cc:2270] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:56.261955 712 leader_election.cc:343] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.262248 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.262274 1157 ts_tablet_manager.cc:1905] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.262542 1155 consensus_queue.cc:260] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.262634 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9f6fbefddf434f32ab373b4e0191cee2 (DEFAULT_TABLE table=default.Table_97c5167bc43d47699812760d5d1939c9 [id=d9e113cbd95c4a809a4adfb8d6c7c8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.262888 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6fbefddf434f32ab373b4e0191cee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.263404 751 tablet_service.cc:1468] Processing CreateTablet for tablet 56b13ca86979406e8d8a56f6e89201f8 (DEFAULT_TABLE table=default.Table_4e7128f0007c44f691cd8bc0931dfe17 [id=76f64c090c6e4d2abd1bc152f42b9011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.263666 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b13ca86979406e8d8a56f6e89201f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.264287 1861 raft_consensus.cc:491] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.264384 1861 raft_consensus.cc:513] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.264819 1861 leader_election.cc:290] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.265688 1155 ts_tablet_manager.cc:1428] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250629 01:51:56.265897 1155 tablet_bootstrap.cc:492] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.267673 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.267959 1020 tablet_service.cc:1468] Processing CreateTablet for tablet a4c5067d924946fabd0b28ab5a68d2ee (DEFAULT_TABLE table=default.Table_f5624ebd8475423285371a2a55453a30 [id=d61c01c2f21f4b0686ba59324fb383f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.268522 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c5067d924946fabd0b28ab5a68d2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.271472 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e0674e2b63de41299b528eacdaf4b93e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.271842 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0674e2b63de41299b528eacdaf4b93e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.273501 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f6fbefddf434f32ab373b4e0191cee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.273976 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1ca655a041b4aff95df14041a51b6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.276032 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d456410cd09845dabd979c123f70741b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.276427 712 leader_election.cc:343] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.278334 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b13ca86979406e8d8a56f6e89201f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.280457 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 53c2c00c5f7b4cefb88523b8f5453c7e (DEFAULT_TABLE table=default.Table_a0699bfad44f4661b365b0924fd37513 [id=c42f1afbcc8e45fda61e26b602b5e89c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.280727 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53c2c00c5f7b4cefb88523b8f5453c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.282945 1157 ts_tablet_manager.cc:1918] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.283317 1157 log.cc:1199] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b0c2dd05f5bb466fbe38ed259ce3a4b8
I20250629 01:51:56.284400 1157 ts_tablet_manager.cc:1939] T b0c2dd05f5bb466fbe38ed259ce3a4b8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.284931 1159 tablet_bootstrap.cc:492] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.285334 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b0c2dd05f5bb466fbe38ed259ce3a4b8 (table default.Table_d7789b77f4814c388e1eb092297120b1 [id=b890fb8e74dc464bbf195cdfa89814be]) successfully deleted
I20250629 01:51:56.286929 1158 ts_tablet_manager.cc:1918] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.287077 1158 log.cc:1199] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:56.287375 1158 ts_tablet_manager.cc:1939] T 54f38ccc485043ba8b8a6f644a3e4f36 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:56.287987 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 1ac60a024a074953b513ae4f60840d25 (DEFAULT_TABLE table=default.Table_b0da5a36b5c141cc848f23ee43a30fa1 [id=1b2e5bc17a524432af71014b105fd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.288307 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ac60a024a074953b513ae4f60840d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.288357 710 leader_election.cc:304] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:56.288682 1861 raft_consensus.cc:2802] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:56.288813 1861 raft_consensus.cc:695] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:56.289016 1861 consensus_queue.cc:237] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.285858 1157 tablet_replica.cc:331] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.295176 1157 raft_consensus.cc:2241] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.295301 1157 raft_consensus.cc:2270] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.295701 1157 ts_tablet_manager.cc:1905] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.298342 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:56.299787 712 consensus_peers.cc:489] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.301236 1157 ts_tablet_manager.cc:1918] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.301278 1155 tablet_bootstrap.cc:654] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.301340 1157 log.cc:1199] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/241a5c9aee3f4297b67b18380ef5ef88
I20250629 01:51:56.301622 1157 ts_tablet_manager.cc:1939] T 241a5c9aee3f4297b67b18380ef5ef88 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.302004 1157 tablet_replica.cc:331] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.302114 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 241a5c9aee3f4297b67b18380ef5ef88 (table default.Table_9b82f19de1c74fcd9973b4a4084712e1 [id=a9617884dddf40ee883c482c170d8a22]) successfully deleted
I20250629 01:51:56.302150 1157 raft_consensus.cc:2241] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.302254 1157 raft_consensus.cc:2270] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.302709 1157 ts_tablet_manager.cc:1905] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.303390 1159 ts_tablet_manager.cc:1397] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.006s
I20250629 01:51:56.304077 1159 raft_consensus.cc:357] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.304252 1159 raft_consensus.cc:383] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.304296 1159 raft_consensus.cc:738] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.304471 1159 consensus_queue.cc:260] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.306928 1159 ts_tablet_manager.cc:1428] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:51:56.314437 1159 tablet_bootstrap.cc:492] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.317201 1159 tablet_bootstrap.cc:654] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.320003 710 leader_election.cc:304] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.322260 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:56.329974 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]) successfully deleted
I20250629 01:51:56.330233 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]) successfully deleted
W20250629 01:51:56.330550 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11b4bdbbf5d3421ab6e3a31de90fff6d already in progress: creating tablet
W20250629 01:51:56.330971 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 556c2050d2cb40f28a3a0bfb9d677d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 556c2050d2cb40f28a3a0bfb9d677d69 already in progress: creating tablet
W20250629 01:51:56.331367 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d54bdcf7e1a14c669472af8acf36aac2 because the tablet was not found. No further retry: Not found: Tablet not found: d54bdcf7e1a14c669472af8acf36aac2
I20250629 01:51:56.337949 1159 tablet_bootstrap.cc:492] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.338053 1159 ts_tablet_manager.cc:1397] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20250629 01:51:56.338757 1159 raft_consensus.cc:357] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.338907 1159 raft_consensus.cc:383] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.338956 1159 raft_consensus.cc:738] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.339149 1159 consensus_queue.cc:260] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.340831 1159 ts_tablet_manager.cc:1428] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:51:56.341046 1159 tablet_bootstrap.cc:492] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.343637 1159 tablet_bootstrap.cc:654] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.343783 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f6fbefddf434f32ab373b4e0191cee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.344523 1157 ts_tablet_manager.cc:1918] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.344612 1157 log.cc:1199] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8799b1858bf2408a85f0956578639393
I20250629 01:51:56.344859 1157 ts_tablet_manager.cc:1939] T 8799b1858bf2408a85f0956578639393 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.348332 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8799b1858bf2408a85f0956578639393 (table default.Table_1b80dba0f8d04043b95a7aaa176636b9 [id=511866fda7ae4bde9cea9777b36cd76f]) successfully deleted
I20250629 01:51:56.352291 1159 tablet_bootstrap.cc:492] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.352397 1159 ts_tablet_manager.cc:1397] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250629 01:51:56.353238 1159 raft_consensus.cc:357] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.353379 1159 raft_consensus.cc:383] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.353427 1159 raft_consensus.cc:738] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.353600 1159 consensus_queue.cc:260] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.355406 1159 ts_tablet_manager.cc:1428] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:56.355201 1158 tablet_replica.cc:331] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:56.360224 1157 tablet_replica.cc:331] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.360373 1157 raft_consensus.cc:2241] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.360486 1157 raft_consensus.cc:2270] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.361013 1157 ts_tablet_manager.cc:1905] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.361255 1158 raft_consensus.cc:2241] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.363453 1154 tablet_bootstrap.cc:492] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.363782 1154 ts_tablet_manager.cc:1397] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.000s
I20250629 01:51:56.365077 1159 tablet_bootstrap.cc:492] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.364351 1158 raft_consensus.cc:2270] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:56.366269 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38252fbce581492fbb3db848a2a31a49 consensus state has no leader
I20250629 01:51:56.367141 1159 tablet_bootstrap.cc:654] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.370162 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 0323947b135a4308903b1eed1ca4db71 (table default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]). Expected version 3 got 0
I20250629 01:51:56.370817 1157 ts_tablet_manager.cc:1918] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.370899 1157 log.cc:1199] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/add482b533d045e7a3010064fae730cf
I20250629 01:51:56.371009 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 65ab6394c32640bbb7c4cd8b77686f32 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:56.371147 1157 ts_tablet_manager.cc:1939] T add482b533d045e7a3010064fae730cf P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.371783 2336 raft_consensus.cc:2747] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.373093 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]) successfully deleted
W20250629 01:51:56.373277 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11b4bdbbf5d3421ab6e3a31de90fff6d already in progress: creating tablet
W20250629 01:51:56.373433 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 556c2050d2cb40f28a3a0bfb9d677d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 556c2050d2cb40f28a3a0bfb9d677d69 already in progress: creating tablet
W20250629 01:51:56.373605 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d54bdcf7e1a14c669472af8acf36aac2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d54bdcf7e1a14c669472af8acf36aac2 already in progress: creating tablet
I20250629 01:51:56.375080 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_978c09358b7649f1b04db780f28564c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.375581 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_978c09358b7649f1b04db780f28564c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:56.375654 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6bf3c7d0bb548e0b77ba8b569a72d0d already in progress: creating tablet
I20250629 01:51:56.375932 1157 tablet_replica.cc:331] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.376127 1157 raft_consensus.cc:2241] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.376269 1157 raft_consensus.cc:2270] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.376869 1157 ts_tablet_manager.cc:1905] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.378266 1159 tablet_bootstrap.cc:492] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.378297 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 897a4cefefe04f06b47070ec7fa1a33b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.378400 1159 ts_tablet_manager.cc:1397] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20250629 01:51:56.378975 1159 raft_consensus.cc:357] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.379153 1159 raft_consensus.cc:383] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.379112 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af755a1cc02841f2a5ec1585c4d9d386"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:56.379244 1159 raft_consensus.cc:738] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.379423 1159 consensus_queue.cc:260] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:56.381201 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 61214us
I20250629 01:51:56.381621 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:56.367184 1052 raft_consensus.cc:1273] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:56.367451 1154 raft_consensus.cc:357] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.383687 1154 raft_consensus.cc:383] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.383793 1154 raft_consensus.cc:738] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.383919 1159 ts_tablet_manager.cc:1428] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:51:56.384240 1154 consensus_queue.cc:260] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.386340 1159 tablet_bootstrap.cc:492] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.386569 1158 ts_tablet_manager.cc:1905] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.387616 1159 tablet_bootstrap.cc:654] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.386947 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af755a1cc02841f2a5ec1585c4d9d386 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:56.392329 710 leader_election.cc:343] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.392475 710 leader_election.cc:304] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.399873 2337 raft_consensus.cc:2747] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.400842 1157 ts_tablet_manager.cc:1918] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.389043 1154 ts_tablet_manager.cc:1428] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250629 01:51:56.401767 1154 tablet_bootstrap.cc:492] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.401844 1159 tablet_bootstrap.cc:492] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.401872 2217 raft_consensus.cc:491] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.401966 2217 raft_consensus.cc:513] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.402453 2217 leader_election.cc:290] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:56.401968 1159 ts_tablet_manager.cc:1397] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250629 01:51:56.400928 1157 log.cc:1199] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8dd084685cc0407da9e12b0a4df386e9
W20250629 01:51:56.403069 850 leader_election.cc:343] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:56.403057 1157 ts_tablet_manager.cc:1939] T 8dd084685cc0407da9e12b0a4df386e9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.404016 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8dd084685cc0407da9e12b0a4df386e9 (table default.Table_875984d711bd4f7da73063fdc6fba264 [id=af75db4dc824485997f324795f65607b]) successfully deleted
W20250629 01:51:56.404536 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ce90c045115844b0b96143f91f1be1d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce90c045115844b0b96143f91f1be1d4 already in progress: creating tablet
I20250629 01:51:56.405982 1154 tablet_bootstrap.cc:654] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.408476 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8873b53d7e42492da64f7f677e05c6f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8873b53d7e42492da64f7f677e05c6f1 already in progress: creating tablet
W20250629 01:51:56.409026 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2e9e6727c5e3422ab1463298cf3ec974 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e9e6727c5e3422ab1463298cf3ec974 already in progress: creating tablet
I20250629 01:51:56.409271 1157 tablet_replica.cc:331] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.409456 1157 raft_consensus.cc:2241] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.409624 1157 raft_consensus.cc:2270] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:56.409855 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c969377aaf034e9b896c1b53366555ff consensus state has no leader
I20250629 01:51:56.410187 1157 ts_tablet_manager.cc:1905] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.413868 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:56.414273 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56b60431b72e4792aa1c32fdee40cdb9" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:56.414613 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d456410cd09845dabd979c123f70741b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:56.414743 1049 raft_consensus.cc:2466] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:56.416409 455 catalog_manager.cc:5582] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:56.416931 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:51:56.419023 1159 raft_consensus.cc:357] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.419159 1159 raft_consensus.cc:383] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.419210 1159 raft_consensus.cc:738] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.419371 1159 consensus_queue.cc:260] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.419770 1159 ts_tablet_manager.cc:1428] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250629 01:51:56.420532 1159 tablet_bootstrap.cc:492] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.421844 1159 tablet_bootstrap.cc:654] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.422199 1154 tablet_bootstrap.cc:492] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.422602 1154 ts_tablet_manager.cc:1397] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20250629 01:51:56.426069 1154 raft_consensus.cc:357] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.427004 1154 raft_consensus.cc:383] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.427430 1154 raft_consensus.cc:738] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.428400 1154 consensus_queue.cc:260] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:56.429412 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2045012c80d049eaabf7dec1df28b087 consensus state has no leader
I20250629 01:51:56.430992 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_84d666c949fc4d9399e1895cc6ed684a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.431380 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84d666c949fc4d9399e1895cc6ed684a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:56.432273 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cde5df0a015744bf804d74c8046ed458 consensus state has no leader
I20250629 01:51:56.432482 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4bd3cf3371774492beb8e01eec5349fc" } modify_external_catalogs: true
I20250629 01:51:56.432583 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4bd3cf3371774492beb8e01eec5349fc" } modify_external_catalogs: true
I20250629 01:51:56.433454 1154 ts_tablet_manager.cc:1428] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:56.433641 1157 ts_tablet_manager.cc:1918] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.433710 1157 log.cc:1199] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:56.433682 1156 ts_tablet_manager.cc:1918] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.433723 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d4e87e759e2f4bf7b03d422364f1bca2" } modify_external_catalogs: true
I20250629 01:51:56.433791 1156 log.cc:1199] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4e354fce951943d69e4c666897f5caac
I20250629 01:51:56.433842 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d4e87e759e2f4bf7b03d422364f1bca2" } modify_external_catalogs: true
I20250629 01:51:56.433960 1157 ts_tablet_manager.cc:1939] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.434062 1156 ts_tablet_manager.cc:1939] T 4e354fce951943d69e4c666897f5caac P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:56.434984 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_97e9884737fa43fe910d121339257f05" } modify_external_catalogs: true
I20250629 01:51:56.434940 1154 tablet_bootstrap.cc:492] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.435093 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_97e9884737fa43fe910d121339257f05" } modify_external_catalogs: true
I20250629 01:51:56.436182 1159 tablet_bootstrap.cc:492] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.436268 1159 ts_tablet_manager.cc:1397] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20250629 01:51:56.436470 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8ad618a93c184b2fbee23eb637589e4a
I20250629 01:51:56.436820 1159 raft_consensus.cc:357] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.436936 1159 raft_consensus.cc:383] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.436975 1159 raft_consensus.cc:738] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.436939 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c5e04a63c4a94fb086aa813f172d054a" } modify_external_catalogs: true
I20250629 01:51:56.437053 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c5e04a63c4a94fb086aa813f172d054a" } modify_external_catalogs: true
I20250629 01:51:56.437132 1159 consensus_queue.cc:260] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.437562 1159 ts_tablet_manager.cc:1428] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250629 01:51:56.437783 850 leader_election.cc:343] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: f8b38c20eb1849b1a9f8acd40315eef5
I20250629 01:51:56.437873 850 leader_election.cc:304] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:56.438005 850 raft_consensus.cc:2684] T f8b38c20eb1849b1a9f8acd40315eef5 P 443c4bd3c5d548ee8cd347d2b4de5293: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:56.439571 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ad618a93c184b2fbee23eb637589e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.440629 1049 raft_consensus.cc:1273] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:56.442582 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3ffcd3aae43d4045b40fb48dd7b5c8a2 consensus state has no leader
W20250629 01:51:56.442688 710 leader_election.cc:343] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2aca26af31ed4bc994f4d62603edc881
I20250629 01:51:56.442770 710 leader_election.cc:304] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.442853 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_727fa08e7ff04debbc63abc9780d6c21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.443233 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_727fa08e7ff04debbc63abc9780d6c21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:56.443274 710 consensus_peers.cc:489] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6bf3c7d0bb548e0b77ba8b569a72d0d. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:51:56.443491 710 leader_election.cc:343] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0ba9eabd06174b87a1485f4921895c5f
I20250629 01:51:56.443567 710 leader_election.cc:304] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.444149 1861 raft_consensus.cc:2747] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.445163 1861 raft_consensus.cc:2747] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.445760 1154 tablet_bootstrap.cc:654] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.446547 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0b847d6ea2594d1880f4d7a9d7e825e8 (DEFAULT_TABLE table=default.Table_11fa9f38439b4f6d990266584eebfaf9 [id=76187ce4b2f349fbb833b92b6ac6f642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.446816 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 65ab6394c32640bbb7c4cd8b77686f32
I20250629 01:51:56.446954 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b847d6ea2594d1880f4d7a9d7e825e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.447409 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 65ab6394c32640bbb7c4cd8b77686f32 with delete_type TABLET_DATA_DELETED (replaced by tablet 16c544420ee242b691671da85406e766) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.447818 454 catalog_manager.cc:6652] ReplaceTablet: tablet 65ab6394c32640bbb7c4cd8b77686f32 deleted and replaced by tablet 16c544420ee242b691671da85406e766
W20250629 01:51:56.449656 988 consensus_peers.cc:489] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.450778 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ad618a93c184b2fbee23eb637589e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.451411 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 56b13ca86979406e8d8a56f6e89201f8 (DEFAULT_TABLE table=default.Table_4e7128f0007c44f691cd8bc0931dfe17 [id=76f64c090c6e4d2abd1bc152f42b9011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.451553 1157 tablet_replica.cc:331] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.451711 1157 raft_consensus.cc:2241] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.451831 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b13ca86979406e8d8a56f6e89201f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.451825 1157 raft_consensus.cc:2270] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.451829 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 65ab6394c32640bbb7c4cd8b77686f32 with delete_type TABLET_DATA_DELETED (replaced by tablet 16c544420ee242b691671da85406e766) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.452102 1159 tablet_bootstrap.cc:492] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.453846 1159 tablet_bootstrap.cc:654] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.453981 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 54f38ccc485043ba8b8a6f644a3e4f36 (table default.Table_119622f658584820a8fcd60b2b640e05 [id=c2c6ad07071540d09fb8e8e13d4e5bea]) successfully deleted
I20250629 01:51:56.455032 1157 ts_tablet_manager.cc:1905] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.456871 1155 tablet_bootstrap.cc:492] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.457063 1155 ts_tablet_manager.cc:1397] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.191s user 0.003s sys 0.002s
I20250629 01:51:56.458209 1155 raft_consensus.cc:357] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.458774 1155 raft_consensus.cc:383] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.458909 1155 raft_consensus.cc:738] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.459254 1155 consensus_queue.cc:260] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.459837 1157 ts_tablet_manager.cc:1918] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.459913 1157 log.cc:1199] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/1ea681bd91e54e4898bcfb72bed6198f
I20250629 01:51:56.459942 1154 tablet_bootstrap.cc:492] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.460122 1154 ts_tablet_manager.cc:1397] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250629 01:51:56.460208 1155 ts_tablet_manager.cc:1428] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:56.461148 1154 raft_consensus.cc:357] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.461405 1154 raft_consensus.cc:383] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.461544 1154 raft_consensus.cc:738] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.461856 1154 consensus_queue.cc:260] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.465463 913 raft_consensus.cc:3058] T 1bb4ff3220cf4d25b680235f1b4882ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.465776 1154 ts_tablet_manager.cc:1428] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:56.468196 1157 ts_tablet_manager.cc:1939] T 1ea681bd91e54e4898bcfb72bed6198f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.469724 1154 tablet_bootstrap.cc:492] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.470019 1155 tablet_bootstrap.cc:492] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.471148 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.472496 1157 tablet_replica.cc:331] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.472640 1157 raft_consensus.cc:2241] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.472836 1157 raft_consensus.cc:2270] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.473371 2347 consensus_queue.cc:1035] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:56.474054 1157 ts_tablet_manager.cc:1905] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:56.475899 710 consensus_peers.cc:489] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: add482b533d045e7a3010064fae730cf. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.476179 1154 tablet_bootstrap.cc:654] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.476265 710 leader_election.cc:343] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.479614 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 65ab6394c32640bbb7c4cd8b77686f32 with delete_type TABLET_DATA_DELETED (replaced by tablet 16c544420ee242b691671da85406e766) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.479828 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ad618a93c184b2fbee23eb637589e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.480084 710 leader_election.cc:304] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.480612 2347 raft_consensus.cc:491] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.480751 2347 raft_consensus.cc:513] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.481330 2347 leader_election.cc:290] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.476408 2343 consensus_queue.cc:1035] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:56.482128 1155 tablet_bootstrap.cc:654] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.482900 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38f831a6dd68493382d5f2920d0794a8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.483461 712 leader_election.cc:343] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:56.484053 710 leader_election.cc:343] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:51:56.485122 2273 consensus_queue.cc:1035] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:56.485877 710 leader_election.cc:304] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.486327 710 leader_election.cc:304] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:56.486944 1157 ts_tablet_manager.cc:1918] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.487035 1157 log.cc:1199] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/897a4cefefe04f06b47070ec7fa1a33b
I20250629 01:51:56.487289 1157 ts_tablet_manager.cc:1939] T 897a4cefefe04f06b47070ec7fa1a33b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.484247 2343 raft_consensus.cc:2747] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.486589 2347 raft_consensus.cc:2747] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.488010 1159 tablet_bootstrap.cc:492] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.488226 1159 ts_tablet_manager.cc:1397] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20250629 01:51:56.488880 1159 raft_consensus.cc:357] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.489059 1159 raft_consensus.cc:383] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.489141 1159 raft_consensus.cc:738] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.489341 1159 consensus_queue.cc:260] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.484544 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1005) took 2110 ms. Trace:
I20250629 01:51:56.489768 1159 ts_tablet_manager.cc:1428] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:56.489982 1159 tablet_bootstrap.cc:492] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.489881 1156 rpcz_store.cc:276] 0629 01:51:54.374027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.374081 (+ 54us) service_pool.cc:225] Handling call
0629 01:51:56.224154 (+1850073us) ts_tablet_manager.cc:1145] Deleting tablet 4e354fce951943d69e4c666897f5caac
0629 01:51:56.224163 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.385807 (+161644us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:56.433601 (+ 47794us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:56.484531 (+ 50930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849866}
I20250629 01:51:56.490530 1156 tablet_replica.cc:331] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:56.490764 1156 raft_consensus.cc:2241] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:56.491216 1159 tablet_bootstrap.cc:654] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.491775 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4e354fce951943d69e4c666897f5caac (table default.Table_67f686028f0c40c991d3231bd0b0c823 [id=c51c68dd349543d4a4a8bfb1b9aa1184]) successfully deleted
I20250629 01:51:56.495640 2273 raft_consensus.cc:491] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.495738 2273 raft_consensus.cc:513] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.496248 2273 leader_election.cc:290] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.496759 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4320f615464fedbcab25137952f684" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.497195 988 leader_election.cc:343] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.498857 1154 tablet_bootstrap.cc:492] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.499069 1154 ts_tablet_manager.cc:1397] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.001s
I20250629 01:51:56.500144 1154 raft_consensus.cc:357] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.500408 1154 raft_consensus.cc:383] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.500542 1154 raft_consensus.cc:738] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.500892 1154 consensus_queue.cc:260] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.501237 1158 ts_tablet_manager.cc:1918] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.502399 1158 log.cc:1199] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/56b60431b72e4792aa1c32fdee40cdb9
I20250629 01:51:56.502974 2217 raft_consensus.cc:491] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.503082 2217 raft_consensus.cc:513] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.503580 2217 leader_election.cc:290] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:56.503872 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:56.509289 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet eb376259f6cd490b87b52cfc1f8e6275
I20250629 01:51:56.510967 2336 raft_consensus.cc:2802] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:56.511082 2336 raft_consensus.cc:491] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:56.511181 2336 raft_consensus.cc:3058] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.511179 751 tablet_service.cc:1515] Processing DeleteTablet for tablet eb376259f6cd490b87b52cfc1f8e6275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.506287 1158 ts_tablet_manager.cc:1939] T 56b60431b72e4792aa1c32fdee40cdb9 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:56.507064 1154 ts_tablet_manager.cc:1428] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250629 01:51:56.513521 2336 raft_consensus.cc:513] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.514053 2336 leader_election.cc:290] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.516278 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 56b60431b72e4792aa1c32fdee40cdb9 (table default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]) successfully deleted
I20250629 01:51:56.516395 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet eb376259f6cd490b87b52cfc1f8e6275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.517220 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0323947b135a4308903b1eed1ca4db71" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:56.517403 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4320f615464fedbcab25137952f684" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:56.517669 850 leader_election.cc:304] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:56.518198 986 leader_election.cc:343] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.518286 986 leader_election.cc:304] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.518471 2273 raft_consensus.cc:2747] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.518640 2217 raft_consensus.cc:2747] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.517377 2347 consensus_queue.cc:1035] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250629 01:51:56.518850 1154 tablet_bootstrap.cc:492] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.519340 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d456410cd09845dabd979c123f70741b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:56.519631 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2e9e6727c5e3422ab1463298cf3ec974 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e9e6727c5e3422ab1463298cf3ec974 already in progress: creating tablet
W20250629 01:51:56.519714 712 leader_election.cc:343] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.520437 1156 raft_consensus.cc:2270] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.520957 1156 ts_tablet_manager.cc:1905] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.521572 1154 tablet_bootstrap.cc:654] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.525419 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38f831a6dd68493382d5f2920d0794a8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:56.525602 1052 raft_consensus.cc:2466] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:56.526520 710 leader_election.cc:304] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:56.526970 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d456410cd09845dabd979c123f70741b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:56.527113 1052 raft_consensus.cc:3058] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.527191 2049 raft_consensus.cc:2802] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:56.527324 2049 raft_consensus.cc:491] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:56.527451 2049 raft_consensus.cc:3058] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.529559 1052 raft_consensus.cc:2466] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:56.531641 710 leader_election.cc:304] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:56.532223 2347 raft_consensus.cc:2802] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:56.532470 2347 raft_consensus.cc:695] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:56.532945 2347 consensus_queue.cc:237] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.536751 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5a1604552122400c8fb0aca541834525
I20250629 01:51:56.538252 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a1604552122400c8fb0aca541834525 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.543365 1154 tablet_bootstrap.cc:492] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.543514 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a1604552122400c8fb0aca541834525 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.543573 1154 ts_tablet_manager.cc:1397] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
W20250629 01:51:56.543711 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 46111.9us
I20250629 01:51:56.544694 1154 raft_consensus.cc:357] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.545220 1154 raft_consensus.cc:383] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.545364 1154 raft_consensus.cc:738] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.545770 1154 consensus_queue.cc:260] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.546502 1154 ts_tablet_manager.cc:1428] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:56.546900 1154 tablet_bootstrap.cc:492] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.550082 1158 tablet_replica.cc:331] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:56.550753 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8873b53d7e42492da64f7f677e05c6f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8873b53d7e42492da64f7f677e05c6f1 already in progress: creating tablet
I20250629 01:51:56.553776 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:56.554527 2347 raft_consensus.cc:491] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.554373 1154 tablet_bootstrap.cc:654] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.554665 2347 raft_consensus.cc:513] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.555398 2347 leader_election.cc:290] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.555611 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54f38ccc485043ba8b8a6f644a3e4f36" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.555990 712 leader_election.cc:343] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.556834 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1d82115081f94493a4e91b0317d43803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.557289 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d82115081f94493a4e91b0317d43803 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.558393 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a4efa38905d0481aaccc078ea261012d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.558758 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4efa38905d0481aaccc078ea261012d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.559755 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 898b49812ece4b57a1a1015d9f26be2f from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:56.562575 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a1604552122400c8fb0aca541834525 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.563292 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f8b38c20eb1849b1a9f8acd40315eef5 (table default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]) successfully deleted
I20250629 01:51:56.563510 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 1ea681bd91e54e4898bcfb72bed6198f (table default.Table_1a2a249712224a0fa06b23867c043cfa [id=2fbd2c29b63c488ba11a1ed546f38da4]) successfully deleted
W20250629 01:51:56.563722 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9f6fbefddf434f32ab373b4e0191cee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f6fbefddf434f32ab373b4e0191cee2 already in progress: creating tablet
I20250629 01:51:56.563916 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 897a4cefefe04f06b47070ec7fa1a33b (table default.Table_5e27d74509cc4dc9a4f0566c61348aa3 [id=374fb405fcde447ea9a2dcb761265caf]) successfully deleted
I20250629 01:51:56.564198 881 tablet_service.cc:1515] Processing DeleteTablet for tablet eb376259f6cd490b87b52cfc1f8e6275 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:56.564568 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8ad618a93c184b2fbee23eb637589e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad618a93c184b2fbee23eb637589e4a already in progress: creating tablet
W20250629 01:51:56.564767 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 65ab6394c32640bbb7c4cd8b77686f32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ab6394c32640bbb7c4cd8b77686f32 already in progress: creating tablet
I20250629 01:51:56.566793 1154 tablet_bootstrap.cc:492] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.566928 1154 ts_tablet_manager.cc:1397] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20250629 01:51:56.567735 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_475fe7b17a5c4aac90faf6432e488d87" } new_table_name: "default.Table_923010b6578e479089f2b7da4e770d9a" modify_external_catalogs: true
I20250629 01:51:56.568019 1154 raft_consensus.cc:357] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.568186 1154 raft_consensus.cc:383] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.568259 1154 raft_consensus.cc:738] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.569305 1154 consensus_queue.cc:260] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.569420 877 tablet_service.cc:1515] Processing DeleteTablet for tablet f5941dd6381041538be6c92858d6ba13 with delete_type TABLET_DATA_DELETED (replaced by tablet 347fa6a12a744ee88058643b7143e39d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.569654 1154 ts_tablet_manager.cc:1428] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:56.569895 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 5c6f3f19a4d441f58c8c6099e04f82ec with delete_type TABLET_DATA_DELETED (replaced by tablet ecbfc4e45dc9435bbd7a55ba7b7e4155) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.570148 1154 tablet_bootstrap.cc:492] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.570492 1157 tablet_replica.cc:331] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.570667 1157 raft_consensus.cc:2241] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.570793 1157 raft_consensus.cc:2270] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.571923 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b4fbc288180d41e2be6faef2985f4b0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.572365 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4fbc288180d41e2be6faef2985f4b0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:56.573184 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 5a1604552122400c8fb0aca541834525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1604552122400c8fb0aca541834525 already in progress: creating tablet
I20250629 01:51:56.573331 1157 ts_tablet_manager.cc:1905] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:56.573415 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet eb376259f6cd490b87b52cfc1f8e6275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb376259f6cd490b87b52cfc1f8e6275 already in progress: creating tablet
I20250629 01:51:56.577795 1155 tablet_bootstrap.cc:492] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.577905 1155 ts_tablet_manager.cc:1397] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.002s
I20250629 01:51:56.578550 1155 raft_consensus.cc:357] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.578613 1157 ts_tablet_manager.cc:1918] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.578681 1155 raft_consensus.cc:383] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.578732 1155 raft_consensus.cc:738] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.578749 1157 log.cc:1199] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f5941dd6381041538be6c92858d6ba13
I20250629 01:51:56.578526 452 catalog_manager.cc:5582] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:56.578981 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:51:56.579048 1157 ts_tablet_manager.cc:1939] T f5941dd6381041538be6c92858d6ba13 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.579227 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
W20250629 01:51:56.579362 710 leader_election.cc:343] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:56.579452 710 leader_election.cc:304] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.579728 1157 tablet_replica.cc:331] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.579888 1157 raft_consensus.cc:2241] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.579998 1157 raft_consensus.cc:2270] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.580344 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3f36a3b189db4f4296fd81d47125b49c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.580700 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f36a3b189db4f4296fd81d47125b49c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.581578 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f5941dd6381041538be6c92858d6ba13 (table default.Table_475fe7b17a5c4aac90faf6432e488d87 [id=0fb1390e9626454eba8e8d5c634e3943]) successfully deleted
I20250629 01:51:56.582079 1157 ts_tablet_manager.cc:1905] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.583844 2049 raft_consensus.cc:513] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.584383 2343 raft_consensus.cc:2747] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.585084 2049 leader_election.cc:290] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.585304 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38f831a6dd68493382d5f2920d0794a8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:56.585673 712 leader_election.cc:343] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:56.588403 712 consensus_peers.cc:489] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.589555 1154 tablet_bootstrap.cc:654] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.590065 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "38f831a6dd68493382d5f2920d0794a8" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:56.590194 1048 raft_consensus.cc:3058] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.590188 2049 raft_consensus.cc:491] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.590278 2049 raft_consensus.cc:513] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.590824 2049 leader_election.cc:290] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.590947 1158 raft_consensus.cc:2241] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.591063 1158 raft_consensus.cc:2270] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.591454 1158 ts_tablet_manager.cc:1905] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.592320 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2845d8fbf3f34c769027b9b6bb3e1f08" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.592664 712 leader_election.cc:343] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.593024 1048 raft_consensus.cc:2466] T 38f831a6dd68493382d5f2920d0794a8 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:56.595083 710 leader_election.cc:304] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:56.595309 2337 raft_consensus.cc:2802] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:56.595418 2337 raft_consensus.cc:695] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:56.595633 2337 consensus_queue.cc:237] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.604748 1155 consensus_queue.cc:260] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.605677 1155 ts_tablet_manager.cc:1428] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20250629 01:51:56.607594 1155 tablet_bootstrap.cc:492] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.610519 1155 tablet_bootstrap.cc:654] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.610589 1159 tablet_bootstrap.cc:492] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.610724 1159 ts_tablet_manager.cc:1397] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.121s user 0.020s sys 0.020s
I20250629 01:51:56.611353 1159 raft_consensus.cc:357] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.611502 1159 raft_consensus.cc:383] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.611555 1159 raft_consensus.cc:738] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.611748 1159 consensus_queue.cc:260] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.612910 1159 ts_tablet_manager.cc:1428] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:56.617178 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ea62c05f0ede4db9b48f76347e23a5cb
I20250629 01:51:56.617784 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_79b81d65f78949388e3fbee231ed031c" } new_table_name: "default.Table_4505da86fcd44db7926a25eef6b827f1" modify_external_catalogs: true
I20250629 01:51:56.618845 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f4a3b7d971694e15be1aaa072468dae2
I20250629 01:51:56.621268 881 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62c05f0ede4db9b48f76347e23a5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.621430 877 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a3b7d971694e15be1aaa072468dae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:56.617813 710 leader_election.cc:343] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2845d8fbf3f34c769027b9b6bb3e1f08
I20250629 01:51:56.621959 710 leader_election.cc:304] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:56.622009 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62c05f0ede4db9b48f76347e23a5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.622197 1157 ts_tablet_manager.cc:1918] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.623899 1157 log.cc:1199] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5c6f3f19a4d441f58c8c6099e04f82ec
I20250629 01:51:56.625139 1159 tablet_bootstrap.cc:492] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.625075 1157 ts_tablet_manager.cc:1939] T 5c6f3f19a4d441f58c8c6099e04f82ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.626420 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b9ef027c9d42048e4a6dff221df793 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.626746 751 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a3b7d971694e15be1aaa072468dae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.627379 2337 raft_consensus.cc:2747] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.632134 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 5c6f3f19a4d441f58c8c6099e04f82ec (table default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]) successfully deleted
I20250629 01:51:56.632917 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 898b49812ece4b57a1a1015d9f26be2f
I20250629 01:51:56.633235 449 catalog_manager.cc:6652] ReplaceTablet: tablet 898b49812ece4b57a1a1015d9f26be2f deleted and replaced by tablet 573b329d020549a08c94891916164aeb
W20250629 01:51:56.633901 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ea62c05f0ede4db9b48f76347e23a5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea62c05f0ede4db9b48f76347e23a5cb already in progress: creating tablet
W20250629 01:51:56.635253 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f4a3b7d971694e15be1aaa072468dae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4a3b7d971694e15be1aaa072468dae2 already in progress: creating tablet
I20250629 01:51:56.636044 1157 tablet_replica.cc:331] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.637121 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 898b49812ece4b57a1a1015d9f26be2f with delete_type TABLET_DATA_DELETED (replaced by tablet 573b329d020549a08c94891916164aeb) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.637490 1157 raft_consensus.cc:2241] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.640132 1157 raft_consensus.cc:2270] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.641609 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e64a7cfad754440186ceb2c9b3fc538e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.642056 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e64a7cfad754440186ceb2c9b3fc538e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.642971 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a8a689877541432b88889c5998bc53fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.643321 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8a689877541432b88889c5998bc53fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.643718 1157 ts_tablet_manager.cc:1905] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.644347 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d11d1c20ae434e399df06213b6217417" } modify_external_catalogs: true
I20250629 01:51:56.644470 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d11d1c20ae434e399df06213b6217417" } modify_external_catalogs: true
I20250629 01:51:56.644711 1155 tablet_bootstrap.cc:492] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.644805 1155 ts_tablet_manager.cc:1397] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20250629 01:51:56.645025 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e068250f05fd4e77bf37790a635a1eb6" } new_table_name: "default.Table_6aad014bd6104b998f834208ad803fdb" modify_external_catalogs: true
I20250629 01:51:56.645424 1155 raft_consensus.cc:357] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.645542 1155 raft_consensus.cc:383] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.645584 1155 raft_consensus.cc:738] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.645941 1155 consensus_queue.cc:260] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.646461 1155 ts_tablet_manager.cc:1428] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:56.646420 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_df0fc31afd154b05a7d3d84bec25e140" } new_table_name: "default.Table_71c6625c776b4ab6a54a4c6d4d928cf2" modify_external_catalogs: true
I20250629 01:51:56.646670 1155 tablet_bootstrap.cc:492] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.647612 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 898b49812ece4b57a1a1015d9f26be2f with delete_type TABLET_DATA_DELETED (replaced by tablet 573b329d020549a08c94891916164aeb) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.652150 1155 tablet_bootstrap.cc:654] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.652829 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet ea62c05f0ede4db9b48f76347e23a5cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.656466 913 raft_consensus.cc:3058] T d159e69137d34899ace5c245638a3c0f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.657893 1155 tablet_bootstrap.cc:492] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.657981 1155 ts_tablet_manager.cc:1397] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250629 01:51:56.658516 1155 raft_consensus.cc:357] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.658638 1155 raft_consensus.cc:383] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.659686 1155 raft_consensus.cc:738] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.659922 1155 consensus_queue.cc:260] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.660463 1155 ts_tablet_manager.cc:1428] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:56.660712 1155 tablet_bootstrap.cc:492] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:56.660729 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 347fa6a12a744ee88058643b7143e39d consensus state has no leader
I20250629 01:51:56.660976 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 9ed1ee73bb8f4761806a66f988b57bf9 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:56.661746 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f798e82e14704a689fe84cbeb9a8d477" } new_table_name: "default.Table_4d28e4ec92114a31856c3992b0728729" modify_external_catalogs: true
I20250629 01:51:56.662297 1155 tablet_bootstrap.cc:654] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.668251 1156 ts_tablet_manager.cc:1918] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:56.668812 1156 log.cc:1199] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/798516e55ca747fcb2942026adb78af0
I20250629 01:51:56.669193 1156 ts_tablet_manager.cc:1939] T 798516e55ca747fcb2942026adb78af0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:56.673342 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a9509620c1c4fcea8fff19dab333f7a has no consensus state
I20250629 01:51:56.679111 1159 tablet_bootstrap.cc:654] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.680655 2357 log.cc:927] Time spent T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250629 01:51:56.682077 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet f4a3b7d971694e15be1aaa072468dae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.682498 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 898b49812ece4b57a1a1015d9f26be2f with delete_type TABLET_DATA_DELETED (replaced by tablet 573b329d020549a08c94891916164aeb) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.687532 1159 tablet_bootstrap.cc:492] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.687682 1159 ts_tablet_manager.cc:1397] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.003s
I20250629 01:51:56.688460 1157 ts_tablet_manager.cc:1918] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.688477 1159 raft_consensus.cc:357] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.689111 1159 raft_consensus.cc:383] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.689240 1159 raft_consensus.cc:738] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.689576 1159 consensus_queue.cc:260] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.676597 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.685703 1155 tablet_bootstrap.cc:492] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.690222 1155 ts_tablet_manager.cc:1397] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20250629 01:51:56.690469 1159 ts_tablet_manager.cc:1428] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:56.690636 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1006) took 2295 ms. Trace:
I20250629 01:51:56.690922 1159 tablet_bootstrap.cc:492] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.692031 1157 log.cc:1199] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/56b9ef027c9d42048e4a6dff221df793
I20250629 01:51:56.693827 1157 ts_tablet_manager.cc:1939] T 56b9ef027c9d42048e4a6dff221df793 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.690790 1155 raft_consensus.cc:357] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.696874 1155 raft_consensus.cc:383] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.696934 1155 raft_consensus.cc:738] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.696799 1156 rpcz_store.cc:276] 0629 01:51:54.395211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.395785 (+ 574us) service_pool.cc:225] Handling call
0629 01:51:56.490483 (+2094698us) ts_tablet_manager.cc:1145] Deleting tablet 798516e55ca747fcb2942026adb78af0
0629 01:51:56.490495 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.594363 (+103868us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:56.623883 (+ 29520us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:56.690621 (+ 66738us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26260,"tablet-delete.queue_time_us":2020677}
I20250629 01:51:56.697723 1159 tablet_bootstrap.cc:654] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.698763 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 56b9ef027c9d42048e4a6dff221df793 (table default.Table_5149d5ed90ca4d6ca10f30e1694a0e1d [id=24d959060a484a2bab405fa65b92abf0]) successfully deleted
W20250629 01:51:56.700430 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 898b49812ece4b57a1a1015d9f26be2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898b49812ece4b57a1a1015d9f26be2f already in progress: creating tablet
I20250629 01:51:56.723661 1158 ts_tablet_manager.cc:1918] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.723763 1158 log.cc:1199] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/c0b9afb9976541b2bc7ab188f35a62ec
I20250629 01:51:56.724030 1158 ts_tablet_manager.cc:1939] T c0b9afb9976541b2bc7ab188f35a62ec P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:56.724498 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet c0b9afb9976541b2bc7ab188f35a62ec (table default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]) successfully deleted
W20250629 01:51:56.724781 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ce90c045115844b0b96143f91f1be1d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce90c045115844b0b96143f91f1be1d4 already in progress: creating tablet
W20250629 01:51:56.724999 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9f6fbefddf434f32ab373b4e0191cee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f6fbefddf434f32ab373b4e0191cee2 already in progress: creating tablet
I20250629 01:51:56.725032 1158 tablet_replica.cc:331] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:56.725193 1158 raft_consensus.cc:2241] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.725322 1158 raft_consensus.cc:2270] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.725843 1158 ts_tablet_manager.cc:1905] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.731109 2374 consensus_queue.cc:1035] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:56.697137 1155 consensus_queue.cc:260] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.700114 2371 tablet.cc:1722] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:56.732446 1155 ts_tablet_manager.cc:1428] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.042s user 0.000s sys 0.001s
I20250629 01:51:56.732685 1155 tablet_bootstrap.cc:492] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.734488 1155 tablet_bootstrap.cc:654] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.738009 1159 tablet_bootstrap.cc:492] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.737958 2336 consensus_queue.cc:1035] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:56.738122 1159 ts_tablet_manager.cc:1397] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.001s
I20250629 01:51:56.738516 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 798516e55ca747fcb2942026adb78af0 (table default.Table_3812c4a8d39844e69e8e142dbf156fae [id=fd0fa3108c2b4816b502364824e4cc12]) successfully deleted
I20250629 01:51:56.738747 1159 raft_consensus.cc:357] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.738884 1159 raft_consensus.cc:383] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.738934 1159 raft_consensus.cc:738] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.739117 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1009) took 2171 ms. Trace:
I20250629 01:51:56.739130 1159 consensus_queue.cc:260] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.739187 1156 rpcz_store.cc:276] 0629 01:51:54.567518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.567574 (+ 56us) service_pool.cc:225] Handling call
0629 01:51:56.738721 (+2171147us) ts_tablet_manager.cc:1145] Deleting tablet 109e19ed0ea64fa4985e93efbaca5107
0629 01:51:56.738727 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.739001 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2170955}
W20250629 01:51:56.739511 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 109e19ed0ea64fa4985e93efbaca5107 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 109e19ed0ea64fa4985e93efbaca5107 already in progress: creating tablet
I20250629 01:51:56.740597 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1010) took 2154 ms. Trace:
I20250629 01:51:56.740909 1156 rpcz_store.cc:276] 0629 01:51:54.585902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.586129 (+ 227us) service_pool.cc:225] Handling call
0629 01:51:56.740510 (+2154381us) ts_tablet_manager.cc:1145] Deleting tablet 8992da4b777d400b9c3f242444229353
0629 01:51:56.740517 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.740582 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2154244}
W20250629 01:51:56.741571 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8992da4b777d400b9c3f242444229353 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8992da4b777d400b9c3f242444229353 already in progress: creating tablet
I20250629 01:51:56.742866 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_02254766c6574c0c991e2621e941a252"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:56.743266 1159 ts_tablet_manager.cc:1428] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20250629 01:51:56.743362 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02254766c6574c0c991e2621e941a252 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.743654 1159 tablet_bootstrap.cc:492] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.741930 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1013) took 2145 ms. Trace:
I20250629 01:51:56.744534 1156 rpcz_store.cc:276] 0629 01:51:54.596358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.596413 (+ 55us) service_pool.cc:225] Handling call
0629 01:51:56.741834 (+2145421us) ts_tablet_manager.cc:1145] Deleting tablet 5614713dab134cb9b6d70c9a53973014
0629 01:51:56.741843 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.741916 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2145254}
I20250629 01:51:56.746176 1159 tablet_bootstrap.cc:654] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.746469 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5614713dab134cb9b6d70c9a53973014 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5614713dab134cb9b6d70c9a53973014 already in progress: creating tablet
I20250629 01:51:56.746825 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1014) took 2111 ms. Trace:
I20250629 01:51:56.747445 1156 rpcz_store.cc:276] 0629 01:51:54.635355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.635536 (+ 181us) service_pool.cc:225] Handling call
0629 01:51:56.746734 (+2111198us) ts_tablet_manager.cc:1145] Deleting tablet 9e0d550773c949df9992be1f9f01b548
0629 01:51:56.746742 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.746803 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111055}
W20250629 01:51:56.748234 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9e0d550773c949df9992be1f9f01b548 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e0d550773c949df9992be1f9f01b548 already in progress: creating tablet
I20250629 01:51:56.750037 1156 tablet_replica.cc:331] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:56.751040 877 tablet_service.cc:1515] Processing DeleteTablet for tablet ef976703d61442f8987ff32070a33592 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.751096 1156 raft_consensus.cc:2241] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.751214 1156 raft_consensus.cc:2270] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.751238 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f6d79ac3c5f04b5db295c93c3aae92cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.751266 1157 tablet_replica.cc:331] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.751452 1157 raft_consensus.cc:2241] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.751567 1157 raft_consensus.cc:2270] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.754568 1157 ts_tablet_manager.cc:1905] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:56.756029 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28da32ab17024f519105c2490efd9136 consensus state has no leader
W20250629 01:51:56.756904 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a43f6594e4844af580c99ea598272aae consensus state has no leader
I20250629 01:51:56.758493 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b60d57f2a4b34062a9ed95ffd636f209" } modify_external_catalogs: true
I20250629 01:51:56.758631 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b60d57f2a4b34062a9ed95ffd636f209" } modify_external_catalogs: true
I20250629 01:51:56.759780 1157 ts_tablet_manager.cc:1918] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.759869 1157 log.cc:1199] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/ef976703d61442f8987ff32070a33592
I20250629 01:51:56.760142 1157 ts_tablet_manager.cc:1939] T ef976703d61442f8987ff32070a33592 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.761044 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet ef976703d61442f8987ff32070a33592 (table default.Table_f220872b43b3433694c2f522f6e54acd [id=b1662df1e42f43eeae59f02a3e446006]) successfully deleted
I20250629 01:51:56.761273 1157 tablet_replica.cc:331] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.761525 1157 raft_consensus.cc:2241] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.761638 1157 raft_consensus.cc:2270] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.761672 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f2fa469b443b4b3e8f3e2d272e328a4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.762189 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2fa469b443b4b3e8f3e2d272e328a4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.762176 1157 ts_tablet_manager.cc:1905] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.762741 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0c1c7d79b0a74e8ea1c1b1f5036283dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.763070 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c1c7d79b0a74e8ea1c1b1f5036283dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.764911 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e0674e2b63de41299b528eacdaf4b93e" } modify_external_catalogs: true
I20250629 01:51:56.765033 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e0674e2b63de41299b528eacdaf4b93e" } modify_external_catalogs: true
I20250629 01:51:56.765774 2371 tablet.cc:1722] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:56.767284 1156 ts_tablet_manager.cc:1905] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:56.768658 988 consensus_peers.cc:489] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:56.770140 1158 ts_tablet_manager.cc:1918] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.770238 1155 tablet_bootstrap.cc:492] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.770342 1155 ts_tablet_manager.cc:1397] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20250629 01:51:56.770980 1155 raft_consensus.cc:357] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.771128 1155 raft_consensus.cc:383] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.771181 1155 raft_consensus.cc:738] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.771282 1158 log.cc:1199] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/20586ba2af9e4cb7adec5ca6c3b76e42
I20250629 01:51:56.771363 1155 consensus_queue.cc:260] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.771755 1155 ts_tablet_manager.cc:1428] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:51:56.771982 1155 tablet_bootstrap.cc:492] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.771787 1158 ts_tablet_manager.cc:1939] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:56.772826 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 20586ba2af9e4cb7adec5ca6c3b76e42 (table default.Table_380dba3955bd458caa9a65e48c00f407 [id=77b6e244496149c5a3e46fdfb5bbe791]) successfully deleted
I20250629 01:51:56.773605 1155 tablet_bootstrap.cc:654] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.775589 1155 tablet_bootstrap.cc:492] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.775740 1155 ts_tablet_manager.cc:1397] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250629 01:51:56.776793 2378 tablet.cc:1722] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:56.776822 1155 raft_consensus.cc:357] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.777032 1155 raft_consensus.cc:383] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.777138 1155 raft_consensus.cc:738] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.777767 2383 tablet.cc:1722] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:56.777678 1155 consensus_queue.cc:260] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.781056 1155 ts_tablet_manager.cc:1428] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250629 01:51:56.784245 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 65ab6394c32640bbb7c4cd8b77686f32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ab6394c32640bbb7c4cd8b77686f32 already in progress: creating tablet
W20250629 01:51:56.784555 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8ad618a93c184b2fbee23eb637589e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad618a93c184b2fbee23eb637589e4a already in progress: creating tablet
W20250629 01:51:56.784832 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet eb376259f6cd490b87b52cfc1f8e6275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb376259f6cd490b87b52cfc1f8e6275 already in progress: creating tablet
W20250629 01:51:56.785212 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 5a1604552122400c8fb0aca541834525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1604552122400c8fb0aca541834525 already in progress: creating tablet
W20250629 01:51:56.785516 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ea62c05f0ede4db9b48f76347e23a5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea62c05f0ede4db9b48f76347e23a5cb already in progress: creating tablet
W20250629 01:51:56.785938 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f4a3b7d971694e15be1aaa072468dae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4a3b7d971694e15be1aaa072468dae2 already in progress: creating tablet
W20250629 01:51:56.786191 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 898b49812ece4b57a1a1015d9f26be2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898b49812ece4b57a1a1015d9f26be2f already in progress: creating tablet
I20250629 01:51:56.786901 1154 tablet_bootstrap.cc:492] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.787041 1154 ts_tablet_manager.cc:1397] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.217s user 0.001s sys 0.009s
I20250629 01:51:56.789278 1154 raft_consensus.cc:357] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.789716 1154 raft_consensus.cc:383] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.789814 1154 raft_consensus.cc:738] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.790148 1154 consensus_queue.cc:260] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.795984 1157 ts_tablet_manager.cc:1918] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.794499 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:56.798336 1154 ts_tablet_manager.cc:1428] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250629 01:51:56.798970 450 catalog_manager.cc:5582] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:56.799435 450 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:56.799563 450 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:56.801218 751 tablet_service.cc:1468] Processing CreateTablet for tablet a22f77869dc847439bbd8ee841983272 (DEFAULT_TABLE table=default.Table_d651acfdb0eb4c21a5301d4b4acd35e7 [id=ad8a559c54fd4724843629f8649d34cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.800750 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a22f77869dc847439bbd8ee841983272 (DEFAULT_TABLE table=default.Table_d651acfdb0eb4c21a5301d4b4acd35e7 [id=ad8a559c54fd4724843629f8649d34cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.804299 1154 tablet_bootstrap.cc:492] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.804322 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22f77869dc847439bbd8ee841983272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.804476 1157 log.cc:1199] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f6d79ac3c5f04b5db295c93c3aae92cb
I20250629 01:51:56.804739 1157 ts_tablet_manager.cc:1939] T f6d79ac3c5f04b5db295c93c3aae92cb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.805488 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f6d79ac3c5f04b5db295c93c3aae92cb (table default.Table_7eb487cf3b984b829ea5456c141ea824 [id=16fca49978904e5c880caa5d56572c22]) successfully deleted
I20250629 01:51:56.807008 1154 tablet_bootstrap.cc:654] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.807796 2068 log.cc:927] Time spent T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Append to log took a long time: real 0.096s user 0.001s sys 0.003s
I20250629 01:51:56.800315 881 tablet_service.cc:1468] Processing CreateTablet for tablet a22f77869dc847439bbd8ee841983272 (DEFAULT_TABLE table=default.Table_d651acfdb0eb4c21a5301d4b4acd35e7 [id=ad8a559c54fd4724843629f8649d34cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.804023 1159 tablet_bootstrap.cc:492] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.808310 1159 ts_tablet_manager.cc:1397] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.011s
I20250629 01:51:56.808434 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22f77869dc847439bbd8ee841983272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.808899 1159 raft_consensus.cc:357] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.809021 1159 raft_consensus.cc:383] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.809288 1159 raft_consensus.cc:738] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.809481 1159 consensus_queue.cc:260] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.809996 1159 ts_tablet_manager.cc:1428] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:56.810315 1159 tablet_bootstrap.cc:492] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.811304 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ab96b385263a4a498312de779e29629c (DEFAULT_TABLE table=default.Table_503924285343420d8e275498070e193b [id=11abbbcb0a0940dea58183f6f553f0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.811573 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab96b385263a4a498312de779e29629c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.812108 1159 tablet_bootstrap.cc:654] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.812546 749 tablet_service.cc:1468] Processing CreateTablet for tablet ab96b385263a4a498312de779e29629c (DEFAULT_TABLE table=default.Table_503924285343420d8e275498070e193b [id=11abbbcb0a0940dea58183f6f553f0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.814585 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9bbbeb43473a4f6f892056a30f62589f with delete_type TABLET_DATA_DELETED (replaced by tablet 290c5ab8364145a39f6ff403ab7b20b1) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.814790 1157 tablet_replica.cc:331] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:56.814918 1157 raft_consensus.cc:2241] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:56.815018 1157 raft_consensus.cc:2270] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.815539 1157 ts_tablet_manager.cc:1905] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.816406 877 tablet_service.cc:1468] Processing CreateTablet for tablet ab96b385263a4a498312de779e29629c (DEFAULT_TABLE table=default.Table_503924285343420d8e275498070e193b [id=11abbbcb0a0940dea58183f6f553f0a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.816704 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab96b385263a4a498312de779e29629c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.818641 886 tablet_service.cc:1468] Processing CreateTablet for tablet dac19d6e87e24041b8eb195c2953ae54 (DEFAULT_TABLE table=default.Table_8f370719dbf6400987554c20446591bc [id=760e7531f20d484e8cf23db2f4c16cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.818908 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac19d6e87e24041b8eb195c2953ae54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.819509 753 tablet_service.cc:1468] Processing CreateTablet for tablet dac19d6e87e24041b8eb195c2953ae54 (DEFAULT_TABLE table=default.Table_8f370719dbf6400987554c20446591bc [id=760e7531f20d484e8cf23db2f4c16cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.820006 1032 tablet_service.cc:1468] Processing CreateTablet for tablet dac19d6e87e24041b8eb195c2953ae54 (DEFAULT_TABLE table=default.Table_8f370719dbf6400987554c20446591bc [id=760e7531f20d484e8cf23db2f4c16cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.820420 890 tablet_service.cc:1468] Processing CreateTablet for tablet 0be335d7fdec49e6b6d1a949f4a28e8c (DEFAULT_TABLE table=default.Table_2fa9db42a28d4da8bb5c062c63e8098b [id=8280850bee3b4a779e5026327717cc11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.820564 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac19d6e87e24041b8eb195c2953ae54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.820688 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be335d7fdec49e6b6d1a949f4a28e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.821259 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0be335d7fdec49e6b6d1a949f4a28e8c (DEFAULT_TABLE table=default.Table_2fa9db42a28d4da8bb5c062c63e8098b [id=8280850bee3b4a779e5026327717cc11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.822001 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2e5b2f5e34904639a37f7a9f85ac7995 (DEFAULT_TABLE table=default.Table_5326881cf36c482ba24c3b952cf65c60 [id=f034e06c31ec45568359b0e3437c6a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.822244 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e5b2f5e34904639a37f7a9f85ac7995. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.822916 756 tablet_service.cc:1468] Processing CreateTablet for tablet 2e5b2f5e34904639a37f7a9f85ac7995 (DEFAULT_TABLE table=default.Table_5326881cf36c482ba24c3b952cf65c60 [id=f034e06c31ec45568359b0e3437c6a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.823559 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0be335d7fdec49e6b6d1a949f4a28e8c (DEFAULT_TABLE table=default.Table_2fa9db42a28d4da8bb5c062c63e8098b [id=8280850bee3b4a779e5026327717cc11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.823805 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be335d7fdec49e6b6d1a949f4a28e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.826817 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2e5b2f5e34904639a37f7a9f85ac7995 (DEFAULT_TABLE table=default.Table_5326881cf36c482ba24c3b952cf65c60 [id=f034e06c31ec45568359b0e3437c6a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.827069 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e5b2f5e34904639a37f7a9f85ac7995. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.828047 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d5e5c0a7d4814ee6895122e2eb2a0717" } modify_external_catalogs: true
I20250629 01:51:56.828230 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d5e5c0a7d4814ee6895122e2eb2a0717" } modify_external_catalogs: true
I20250629 01:51:56.828389 886 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9509620c1c4fcea8fff19dab333f7a (DEFAULT_TABLE table=default.Table_4505da86fcd44db7926a25eef6b827f1 [id=5b8dfcc2c441421e8342c016309fca96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.828866 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9509620c1c4fcea8fff19dab333f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.831595 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:56.833660 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 20958c5e88454e129aa8efefb63b7f14
I20250629 01:51:56.835032 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 20958c5e88454e129aa8efefb63b7f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.836030 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 20958c5e88454e129aa8efefb63b7f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:56.837491 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 20958c5e88454e129aa8efefb63b7f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.840335 1158 tablet_replica.cc:331] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:56.840516 1158 raft_consensus.cc:2241] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:56.840755 1158 raft_consensus.cc:2270] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:56.841193 1158 ts_tablet_manager.cc:1905] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:56.843540 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_23573af00cf74adb8eee3267f019a3ee" } modify_external_catalogs: true
I20250629 01:51:56.843667 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_23573af00cf74adb8eee3267f019a3ee" } modify_external_catalogs: true
I20250629 01:51:56.847781 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9509620c1c4fcea8fff19dab333f7a (DEFAULT_TABLE table=default.Table_4505da86fcd44db7926a25eef6b827f1 [id=5b8dfcc2c441421e8342c016309fca96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.848073 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9509620c1c4fcea8fff19dab333f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.849141 747 tablet_service.cc:1468] Processing CreateTablet for tablet 6a9509620c1c4fcea8fff19dab333f7a (DEFAULT_TABLE table=default.Table_4505da86fcd44db7926a25eef6b827f1 [id=5b8dfcc2c441421e8342c016309fca96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.850694 1158 ts_tablet_manager.cc:1918] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:56.850780 1158 log.cc:1199] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/20958c5e88454e129aa8efefb63b7f14
I20250629 01:51:56.851112 1158 ts_tablet_manager.cc:1939] T 20958c5e88454e129aa8efefb63b7f14 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:56.852663 1159 tablet_bootstrap.cc:492] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:56.852763 1159 ts_tablet_manager.cc:1397] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250629 01:51:56.853018 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 6324c387cc474a44bfb215d8345baf16
I20250629 01:51:56.853281 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d96f54aeed554f3bbe79066de62debc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:56.853330 1159 raft_consensus.cc:357] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.853449 1159 raft_consensus.cc:383] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.853493 1159 raft_consensus.cc:738] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:51:56.853693 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d96f54aeed554f3bbe79066de62debc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.853649 1159 consensus_queue.cc:260] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:56.855894 2363 raft_consensus.cc:491] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.856033 2363 raft_consensus.cc:513] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.856753 2363 leader_election.cc:290] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.857285 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:56.857767 986 leader_election.cc:343] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.858875 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.859615 988 leader_election.cc:343] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.859715 988 leader_election.cc:304] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:56.860083 2363 raft_consensus.cc:2747] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:56.860919 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 24c4841e127b495cbfbca0103662c01e (DEFAULT_TABLE table=default.Table_a08dd1de5190465b9b899ba5eaf80fff [id=b962d2c3e8df427385963a446a53ded5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.861193 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c4841e127b495cbfbca0103662c01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.862836 744 tablet_service.cc:1468] Processing CreateTablet for tablet 24c4841e127b495cbfbca0103662c01e (DEFAULT_TABLE table=default.Table_a08dd1de5190465b9b899ba5eaf80fff [id=b962d2c3e8df427385963a446a53ded5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.863687 890 tablet_service.cc:1468] Processing CreateTablet for tablet 1a1bd40bd9794db08dd860cdb12211e1 (DEFAULT_TABLE table=default.Table_0818e53ec2ef4adeba03cabe3d5dfcc5 [id=7266dc33cbf9426598789a417cafcb33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.863976 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1bd40bd9794db08dd860cdb12211e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.870357 1157 ts_tablet_manager.cc:1918] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:56.870443 1157 log.cc:1199] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9bbbeb43473a4f6f892056a30f62589f
I20250629 01:51:56.870672 1157 ts_tablet_manager.cc:1939] T 9bbbeb43473a4f6f892056a30f62589f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:56.873570 1154 tablet_bootstrap.cc:492] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.873713 1154 ts_tablet_manager.cc:1397] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20250629 01:51:56.874338 1154 raft_consensus.cc:357] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.874539 1154 raft_consensus.cc:383] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.874629 1154 raft_consensus.cc:738] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.874850 1154 consensus_queue.cc:260] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.877202 1154 ts_tablet_manager.cc:1428] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:51:56.877583 1154 tablet_bootstrap.cc:492] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.878978 1154 tablet_bootstrap.cc:654] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.882474 886 tablet_service.cc:1468] Processing CreateTablet for tablet 24c4841e127b495cbfbca0103662c01e (DEFAULT_TABLE table=default.Table_a08dd1de5190465b9b899ba5eaf80fff [id=b962d2c3e8df427385963a446a53ded5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.883002 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c4841e127b495cbfbca0103662c01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.886044 1155 tablet_bootstrap.cc:492] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:56.886705 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22f77869dc847439bbd8ee841983272. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.886904 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e5b2f5e34904639a37f7a9f85ac7995. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.887044 2337 raft_consensus.cc:491] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.887248 2337 raft_consensus.cc:513] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.892485 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0be335d7fdec49e6b6d1a949f4a28e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.892683 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a9509620c1c4fcea8fff19dab333f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.893081 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac19d6e87e24041b8eb195c2953ae54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.893676 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24c4841e127b495cbfbca0103662c01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.893926 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab96b385263a4a498312de779e29629c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.899955 1155 tablet_bootstrap.cc:654] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.903479 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6a012ba891949538800581436ef5380" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.903985 712 leader_election.cc:343] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.903048 2337 leader_election.cc:290] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.924891 2374 raft_consensus.cc:491] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.925036 2374 raft_consensus.cc:513] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.925645 2374 leader_election.cc:290] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.926205 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2249d5cba1544653b088312e7d74802e" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:56.926383 775 raft_consensus.cc:2466] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:56.926884 986 leader_election.cc:304] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:56.927181 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2249d5cba1544653b088312e7d74802e" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.927564 988 leader_election.cc:343] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.927811 2374 raft_consensus.cc:2802] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:56.927891 2374 raft_consensus.cc:491] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:56.927942 2374 raft_consensus.cc:3058] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.927960 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7e373110e0024c6dbab7e7b4039555a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.928452 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e373110e0024c6dbab7e7b4039555a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.930220 2374 raft_consensus.cc:513] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.930918 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2249d5cba1544653b088312e7d74802e" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:56.931046 775 raft_consensus.cc:3058] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:56.931773 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2249d5cba1544653b088312e7d74802e" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:56.932168 988 leader_election.cc:343] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.932283 2374 leader_election.cc:290] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.934048 886 tablet_service.cc:1468] Processing CreateTablet for tablet b0daa93d5b22407796027ba8740026e6 (DEFAULT_TABLE table=default.Table_3fce5fefd74344c29e1895554190fd44 [id=58912637f5a5431293475d0d6d10025b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.934370 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0daa93d5b22407796027ba8740026e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.934635 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5e61dd17877340b49c38eafe0498397a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.935014 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e61dd17877340b49c38eafe0498397a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.936046 775 raft_consensus.cc:2466] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:56.936054 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a092a591616041599f1b07e758ef38d1" } new_table_name: "default.Table_95d463e5367e48de86475773e298e184" modify_external_catalogs: true
I20250629 01:51:56.936372 986 leader_election.cc:304] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:56.936651 2374 raft_consensus.cc:2802] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:56.937568 2374 raft_consensus.cc:695] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:56.937774 2374 consensus_queue.cc:237] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.938836 748 tablet_service.cc:1468] Processing CreateTablet for tablet 1a1bd40bd9794db08dd860cdb12211e1 (DEFAULT_TABLE table=default.Table_0818e53ec2ef4adeba03cabe3d5dfcc5 [id=7266dc33cbf9426598789a417cafcb33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.939158 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1bd40bd9794db08dd860cdb12211e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.939888 1154 tablet_bootstrap.cc:492] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.939991 1154 ts_tablet_manager.cc:1397] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.001s
I20250629 01:51:56.940618 1154 raft_consensus.cc:357] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.940740 1154 raft_consensus.cc:383] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.940786 1154 raft_consensus.cc:738] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.940945 1154 consensus_queue.cc:260] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.941357 1154 ts_tablet_manager.cc:1428] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:56.941375 2337 raft_consensus.cc:491] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:56.941460 2337 raft_consensus.cc:513] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.941581 1154 tablet_bootstrap.cc:492] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.942682 1154 tablet_bootstrap.cc:654] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.944447 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47cbc958eca14ed69fb45c3c2984aeef" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:56.945050 712 leader_election.cc:343] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:56.947136 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 33aa0a66cdbf49c5820e2b43b021f28b
I20250629 01:51:56.947676 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_bac468da747c48e4bf6c7884bbd9e1f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.948050 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bac468da747c48e4bf6c7884bbd9e1f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.949875 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:56.950019 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dd06d636d7c6466599c047ba9ccbde72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.950353 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd06d636d7c6466599c047ba9ccbde72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.951776 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e5e8b8e4d42f43c78f34102b76b4b11a (table default.Table_904b8b7a3ab1414a9bd29274648bbb86 [id=e991e29b3d9d4dcc96912754565336f8]). Expected version 1 got 0
I20250629 01:51:56.951915 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 82a837932f14419c8b32fcd2c9358215 (table default.Table_3592b5e7e0c44766a622a04a3a5c9b03 [id=ed3c8fa707924f8ea53d590a7f9b3d68]). Expected version 2 got 0
I20250629 01:51:56.952026 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_f798e82e14704a689fe84cbeb9a8d477 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 4 got 0
I20250629 01:51:56.952208 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:56.953192 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_001859f907034195a0b9358f6fff60c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:56.953574 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_001859f907034195a0b9358f6fff60c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:56.961287 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:56.961778 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]). Expected version 1 got 0
I20250629 01:51:56.962069 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:56.962347 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_4d28e4ec92114a31856c3992b0728729 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 5 got 4
I20250629 01:51:56.962533 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943]). Expected version 5 got 0
I20250629 01:51:56.964376 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.964823 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4320f615464fedbcab25137952f684 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.965703 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:56.965891 1159 ts_tablet_manager.cc:1428] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.113s user 0.015s sys 0.002s
I20250629 01:51:56.966535 2337 leader_election.cc:290] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:56.967216 1159 tablet_bootstrap.cc:492] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:56.973187 1155 tablet_bootstrap.cc:492] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:56.973348 1155 ts_tablet_manager.cc:1397] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.005s
I20250629 01:51:56.974308 1155 raft_consensus.cc:357] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.974509 1155 raft_consensus.cc:383] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.974584 1155 raft_consensus.cc:738] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.974848 1155 consensus_queue.cc:260] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.975389 1155 ts_tablet_manager.cc:1428] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:56.975827 2371 tablet.cc:1717] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Skipping requested alter to schema version 4, tablet already version 4
I20250629 01:51:56.976142 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]) successfully deleted
I20250629 01:51:56.978217 1154 tablet_bootstrap.cc:492] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.978334 1154 ts_tablet_manager.cc:1397] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.001s
I20250629 01:51:56.978636 886 tablet_service.cc:1468] Processing CreateTablet for tablet ff26f8ed91be432fb708a9dad260b30d (DEFAULT_TABLE table=default.Table_54c1b81ff73b436fbf764565c0ab55ca [id=f55f6cf2078e42dd843fb07a85afcc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.978935 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff26f8ed91be432fb708a9dad260b30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:56.978971 1154 raft_consensus.cc:357] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.979100 1154 raft_consensus.cc:383] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.979152 1154 raft_consensus.cc:738] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.979295 1154 consensus_queue.cc:260] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.979686 1154 ts_tablet_manager.cc:1428] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:56.979908 1154 tablet_bootstrap.cc:492] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.981314 747 tablet_service.cc:1468] Processing CreateTablet for tablet b0daa93d5b22407796027ba8740026e6 (DEFAULT_TABLE table=default.Table_3fce5fefd74344c29e1895554190fd44 [id=58912637f5a5431293475d0d6d10025b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:56.983608 1154 tablet_bootstrap.cc:654] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:56.990459 440 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet 347fa6a12a744ee88058643b7143e39d consensus state has no leader
I20250629 01:51:56.990752 1159 tablet_bootstrap.cc:654] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:56.992854 1154 tablet_bootstrap.cc:492] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:56.993009 1154 ts_tablet_manager.cc:1397] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250629 01:51:56.993594 1154 raft_consensus.cc:357] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.993765 1154 raft_consensus.cc:383] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:56.993841 1154 raft_consensus.cc:738] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:56.994017 1154 consensus_queue.cc:260] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:56.994428 1154 ts_tablet_manager.cc:1428] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:56.996290 1154 tablet_bootstrap.cc:492] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:56.997380 1154 tablet_bootstrap.cc:654] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.004987 744 tablet_service.cc:1468] Processing CreateTablet for tablet ff26f8ed91be432fb708a9dad260b30d (DEFAULT_TABLE table=default.Table_54c1b81ff73b436fbf764565c0ab55ca [id=f55f6cf2078e42dd843fb07a85afcc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.005462 890 tablet_service.cc:1468] Processing CreateTablet for tablet 72b2caf969db48458ea007ae03a0640f (DEFAULT_TABLE table=default.Table_78bb87e44b9e4ceea207030a95646888 [id=6c973b28cf4a412e8e9a6227fc9f53f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.005769 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72b2caf969db48458ea007ae03a0640f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.006336 749 tablet_service.cc:1468] Processing CreateTablet for tablet 72b2caf969db48458ea007ae03a0640f (DEFAULT_TABLE table=default.Table_78bb87e44b9e4ceea207030a95646888 [id=6c973b28cf4a412e8e9a6227fc9f53f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.006947 877 tablet_service.cc:1468] Processing CreateTablet for tablet 637b5cbd81454ee6bb5be6f353000551 (DEFAULT_TABLE table=default.Table_45100053e91845d1b945caac267d7349 [id=5ecf7e92979e485191b42ef83ce7d6d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.007292 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637b5cbd81454ee6bb5be6f353000551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.013041 2374 raft_consensus.cc:491] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.013155 2374 raft_consensus.cc:513] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.013865 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "258994dd24804f4aaa5e299436fac9b0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:57.014271 986 leader_election.cc:343] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.014595 1155 tablet_bootstrap.cc:492] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.015722 1155 tablet_bootstrap.cc:654] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.016806 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "258994dd24804f4aaa5e299436fac9b0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.017247 988 leader_election.cc:343] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.017340 988 leader_election.cc:304] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.017536 2363 raft_consensus.cc:2747] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.017675 2374 leader_election.cc:290] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.020080 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0daa93d5b22407796027ba8740026e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.021179 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9bbbeb43473a4f6f892056a30f62589f (table default.Table_dc8c2079eef0455a8ba102bd39414e15 [id=8705c82c65df415abf3299b10b90f48c]) successfully deleted
I20250629 01:51:57.021874 1157 tablet_replica.cc:331] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:57.021879 1159 tablet_bootstrap.cc:492] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.022025 1157 raft_consensus.cc:2241] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.022148 1157 raft_consensus.cc:2270] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.022318 1159 ts_tablet_manager.cc:1397] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250629 01:51:57.022626 1157 ts_tablet_manager.cc:1905] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.024351 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff26f8ed91be432fb708a9dad260b30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.024433 1156 ts_tablet_manager.cc:1918] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.024519 1156 log.cc:1199] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:57.024816 1156 ts_tablet_manager.cc:1939] T 3db333c60cb3485a9daae316b9f1a8c8 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:57.025039 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 69597.4us
I20250629 01:51:57.025043 1159 raft_consensus.cc:357] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.025701 1159 raft_consensus.cc:383] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.026010 1159 raft_consensus.cc:738] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.026834 1159 consensus_queue.cc:260] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.028582 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72b2caf969db48458ea007ae03a0640f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.032051 1159 ts_tablet_manager.cc:1428] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250629 01:51:57.032155 753 tablet_service.cc:1468] Processing CreateTablet for tablet 637b5cbd81454ee6bb5be6f353000551 (DEFAULT_TABLE table=default.Table_45100053e91845d1b945caac267d7349 [id=5ecf7e92979e485191b42ef83ce7d6d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.032500 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637b5cbd81454ee6bb5be6f353000551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.036317 1159 tablet_bootstrap.cc:492] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.039778 1155 tablet_bootstrap.cc:492] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.039891 1155 ts_tablet_manager.cc:1397] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.005s
I20250629 01:51:57.040496 1155 raft_consensus.cc:357] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.040624 1155 raft_consensus.cc:383] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.040668 1155 raft_consensus.cc:738] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.040814 1155 consensus_queue.cc:260] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.041237 1155 ts_tablet_manager.cc:1428] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:57.044292 1155 tablet_bootstrap.cc:492] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.045403 1155 tablet_bootstrap.cc:654] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.045531 1159 tablet_bootstrap.cc:654] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.047319 2374 raft_consensus.cc:491] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.047430 2374 raft_consensus.cc:513] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.047955 2374 leader_election.cc:290] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.048264 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1021) took 2226 ms. Trace:
I20250629 01:51:57.048331 1156 rpcz_store.cc:276] 0629 01:51:54.822212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.822265 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:56.749989 (+1927724us) ts_tablet_manager.cc:1145] Deleting tablet 3db333c60cb3485a9daae316b9f1a8c8
0629 01:51:56.749999 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:56.891271 (+141272us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.024343 (+133072us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.048251 (+ 23908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927571}
I20250629 01:51:57.048403 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:57.048534 775 raft_consensus.cc:2466] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:57.048673 1156 tablet_replica.cc:331] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.048820 1156 raft_consensus.cc:2241] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.048944 1156 raft_consensus.cc:2270] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.049278 986 leader_election.cc:304] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:57.049330 1156 ts_tablet_manager.cc:1905] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.048998 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.049465 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 962f1ac223074f11a69061a69396b3aa
I20250629 01:51:57.049678 2374 raft_consensus.cc:2802] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250629 01:51:57.049702 988 leader_election.cc:343] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.049743 2374 raft_consensus.cc:491] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:57.049793 2374 raft_consensus.cc:3058] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.050725 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet ddc24d0e668c49ac825b2578849ab2a3 (table default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e]). Expected version 2 got 0
I20250629 01:51:57.051991 2374 raft_consensus.cc:513] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.052258 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6b0ef1ba46ac487ab1502c3d116814e1
I20250629 01:51:57.052708 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:57.052825 775 raft_consensus.cc:3058] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.054895 775 raft_consensus.cc:2466] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:57.056236 986 leader_election.cc:304] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:57.056429 2363 raft_consensus.cc:2802] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:57.056504 2363 raft_consensus.cc:695] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:57.056681 2363 consensus_queue.cc:237] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.058245 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 3db333c60cb3485a9daae316b9f1a8c8 (table default.Table_f81f54577f464e6bbe68673617d0e0cd [id=d929c72226424c82a6d8c8bac1c3aac0]) successfully deleted
I20250629 01:51:57.058905 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b0ef1ba46ac487ab1502c3d116814e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.059158 2374 leader_election.cc:290] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.059458 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:57.059793 988 leader_election.cc:343] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.060781 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 962f1ac223074f11a69061a69396b3aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.063858 2217 raft_consensus.cc:491] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.063953 2217 raft_consensus.cc:513] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.064422 2217 leader_election.cc:290] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:57.064373 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 1a1bd40bd9794db08dd860cdb12211e1 (DEFAULT_TABLE table=default.Table_0818e53ec2ef4adeba03cabe3d5dfcc5 [id=7266dc33cbf9426598789a417cafcb33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.064674 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1bd40bd9794db08dd860cdb12211e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:57.065335 850 leader_election.cc:343] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:57.065369 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet ddc24d0e668c49ac825b2578849ab2a3 (table default.Table_e598c56d6dcd40628668325002186227 [id=d3d515038aa84669a3a64824de88f06e]). Expected version 2 got 0
I20250629 01:51:57.066121 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b5e3cea3134a4dab8aa3e9b75293d5de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:57.066845 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e4309edc8dd64fc88947c246a80faac7" } new_table_name: "default.Table_96b34d69a5bc436f8f279e9dd6c00f62" modify_external_catalogs: true
I20250629 01:51:57.067965 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0d73122281254c95b722afbfdd68d670" } new_table_name: "default.Table_9c7f015311024589a1da970cdda15955" modify_external_catalogs: true
W20250629 01:51:57.068454 850 leader_election.cc:343] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:57.068543 850 leader_election.cc:304] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.068897 2217 raft_consensus.cc:2747] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.069521 890 tablet_service.cc:1515] Processing DeleteTablet for tablet cabb0298771e4fd5be29e530d3a8ed3d with delete_type TABLET_DATA_DELETED (replaced by tablet 4aa8dfd712c043ad9f8ff56b064bb8fd) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.072340 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:57.072561 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:57.072752 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_4d28e4ec92114a31856c3992b0728729 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 5 got 4
I20250629 01:51:57.072934 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
I20250629 01:51:57.073213 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:57.073237 753 tablet_service.cc:1468] Processing CreateTablet for tablet 1e87caeaa536423685a228252bb9a901 (DEFAULT_TABLE table=default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4 [id=31aa3535cfa04df18e84dfef0b4c084e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.073382 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]). Expected version 1 got 0
I20250629 01:51:57.073541 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e87caeaa536423685a228252bb9a901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.074492 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b0ef1ba46ac487ab1502c3d116814e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.077431 1159 tablet_bootstrap.cc:492] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.077638 1159 ts_tablet_manager.cc:1397] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250629 01:51:57.078454 1159 raft_consensus.cc:357] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.078940 877 tablet_service.cc:1468] Processing CreateTablet for tablet 1e87caeaa536423685a228252bb9a901 (DEFAULT_TABLE table=default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4 [id=31aa3535cfa04df18e84dfef0b4c084e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.078965 890 tablet_service.cc:1468] Processing CreateTablet for tablet a2c4662f83d64bf896e58df3a6a5e140 (DEFAULT_TABLE table=default.Table_6d58bedddad3413d9c67e6aa0caaa6cb [id=11096ffbe13a4e5faa0864175d419d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.078698 1159 raft_consensus.cc:383] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.079417 1159 raft_consensus.cc:738] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.079594 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c4662f83d64bf896e58df3a6a5e140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.079728 749 tablet_service.cc:1468] Processing CreateTablet for tablet a2c4662f83d64bf896e58df3a6a5e140 (DEFAULT_TABLE table=default.Table_6d58bedddad3413d9c67e6aa0caaa6cb [id=11096ffbe13a4e5faa0864175d419d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.079746 1159 consensus_queue.cc:260] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.080175 881 tablet_service.cc:1468] Processing CreateTablet for tablet 4fbb7bbd7974494d8240b8ecf2900ac5 (DEFAULT_TABLE table=default.Table_739fd1ba816f484ab8a493fe8703336f [id=9b2cbfd0bf5c48be8c1640090ca60ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.080734 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbb7bbd7974494d8240b8ecf2900ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.081724 1159 ts_tablet_manager.cc:1428] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:57.084291 884 tablet_service.cc:1468] Processing CreateTablet for tablet aee0ea5e688c492885da7018e65d93e8 (DEFAULT_TABLE table=default.Table_5d856d8a80084e8bae95e734dff72636 [id=eebe2b05591547f3b58db56c4ade2216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.084720 747 tablet_service.cc:1468] Processing CreateTablet for tablet 4fbb7bbd7974494d8240b8ecf2900ac5 (DEFAULT_TABLE table=default.Table_739fd1ba816f484ab8a493fe8703336f [id=9b2cbfd0bf5c48be8c1640090ca60ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.084815 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.085116 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb with delete_type TABLET_DATA_DELETED (replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.085467 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4320f615464fedbcab25137952f684 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.085604 1159 tablet_bootstrap.cc:492] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.085474 1158 tablet_replica.cc:331] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.073855 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5d60769b6bad4b8eb8dfa90c3047c054" } new_table_name: "default.Table_804afe1b3b414511bd56508601f08895" modify_external_catalogs: true
I20250629 01:51:57.086378 1158 raft_consensus.cc:2241] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.086505 1158 raft_consensus.cc:2270] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.086951 1158 ts_tablet_manager.cc:1905] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:57.074306 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5e3cea3134a4dab8aa3e9b75293d5de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.087261 1159 tablet_bootstrap.cc:654] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.087796 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b0daa93d5b22407796027ba8740026e6 (DEFAULT_TABLE table=default.Table_3fce5fefd74344c29e1895554190fd44 [id=58912637f5a5431293475d0d6d10025b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.088265 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0daa93d5b22407796027ba8740026e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.091497 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 962f1ac223074f11a69061a69396b3aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.095257 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 72b2caf969db48458ea007ae03a0640f (DEFAULT_TABLE table=default.Table_78bb87e44b9e4ceea207030a95646888 [id=6c973b28cf4a412e8e9a6227fc9f53f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.095551 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72b2caf969db48458ea007ae03a0640f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.097215 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aee0ea5e688c492885da7018e65d93e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.100299 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e87caeaa536423685a228252bb9a901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.102284 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 637b5cbd81454ee6bb5be6f353000551 (DEFAULT_TABLE table=default.Table_45100053e91845d1b945caac267d7349 [id=5ecf7e92979e485191b42ef83ce7d6d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.108134 1158 ts_tablet_manager.cc:1918] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.108229 1158 log.cc:1199] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:51:57.108525 1158 ts_tablet_manager.cc:1939] T cb7314fbdfc74b1c8d9d1be9c05244cb P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:57.108611 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637b5cbd81454ee6bb5be6f353000551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.109311 1158 tablet_replica.cc:331] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.109481 1158 raft_consensus.cc:2241] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.109596 1158 raft_consensus.cc:2270] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.110060 1158 ts_tablet_manager.cc:1905] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.118257 1155 tablet_bootstrap.cc:492] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.118413 1155 ts_tablet_manager.cc:1397] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.002s
I20250629 01:51:57.119041 1155 raft_consensus.cc:357] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.119232 1155 raft_consensus.cc:383] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.119319 1155 raft_consensus.cc:738] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.119532 1155 consensus_queue.cc:260] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.120011 1155 ts_tablet_manager.cc:1428] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:57.120397 1155 tablet_bootstrap.cc:492] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.121467 1155 tablet_bootstrap.cc:654] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.124488 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c4662f83d64bf896e58df3a6a5e140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.125216 1157 ts_tablet_manager.cc:1918] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:57.125303 1157 log.cc:1199] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/20958c5e88454e129aa8efefb63b7f14
I20250629 01:51:57.128225 753 tablet_service.cc:1468] Processing CreateTablet for tablet aee0ea5e688c492885da7018e65d93e8 (DEFAULT_TABLE table=default.Table_5d856d8a80084e8bae95e734dff72636 [id=eebe2b05591547f3b58db56c4ade2216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.128279 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbb7bbd7974494d8240b8ecf2900ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.130820 877 tablet_service.cc:1468] Processing CreateTablet for tablet 74f3e7bf715c4032950ac095cb9b2e75 (DEFAULT_TABLE table=default.Table_b913ec7dcfc14a73ae38f883f9e5296d [id=ad32e1cae81849b795b21e0026a5cba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.131178 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f3e7bf715c4032950ac095cb9b2e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.131888 884 tablet_service.cc:1468] Processing CreateTablet for tablet c4c4153b6dcd45ada716c05d0590d10a (DEFAULT_TABLE table=default.Table_2f55e3c1a16e4082aaacb9434d6d583c [id=4718972c19d94c1bb83aee42b1768a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.132172 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c4153b6dcd45ada716c05d0590d10a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:57.133328 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5160fe759e074d93872a8f0ca7847299 consensus state has no leader
I20250629 01:51:57.133630 749 tablet_service.cc:1468] Processing CreateTablet for tablet 74f3e7bf715c4032950ac095cb9b2e75 (DEFAULT_TABLE table=default.Table_b913ec7dcfc14a73ae38f883f9e5296d [id=ad32e1cae81849b795b21e0026a5cba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.133996 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f3e7bf715c4032950ac095cb9b2e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.134935 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 49844d26db6447de8c323376c7756437 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:57.135890 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 4e354fce951943d69e4c666897f5caac with delete_type TABLET_DATA_DELETED (replaced by tablet 3a81e6f9711c4d398ae09f33a75566ae) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.136266 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aee0ea5e688c492885da7018e65d93e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.139066 916 raft_consensus.cc:3058] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.139107 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 1e87caeaa536423685a228252bb9a901 (DEFAULT_TABLE table=default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4 [id=31aa3535cfa04df18e84dfef0b4c084e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.139360 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e87caeaa536423685a228252bb9a901. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.142486 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 962f1ac223074f11a69061a69396b3aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.147289 2374 raft_consensus.cc:491] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.147329 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4fbb7bbd7974494d8240b8ecf2900ac5 (DEFAULT_TABLE table=default.Table_739fd1ba816f484ab8a493fe8703336f [id=9b2cbfd0bf5c48be8c1640090ca60ee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.147397 1155 tablet_bootstrap.cc:492] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.147406 2374 raft_consensus.cc:513] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.147539 1155 ts_tablet_manager.cc:1397] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
I20250629 01:51:57.147661 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbb7bbd7974494d8240b8ecf2900ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.148187 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "daf312e60c014fe698d3ec9f1a1b769c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:57.148530 2374 leader_election.cc:290] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.148559 1155 raft_consensus.cc:357] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.148758 1155 raft_consensus.cc:383] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.148837 1155 raft_consensus.cc:738] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
W20250629 01:51:57.148836 986 leader_election.cc:343] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.149111 1155 consensus_queue.cc:260] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.149796 1155 ts_tablet_manager.cc:1428] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:57.151014 1155 tablet_bootstrap.cc:492] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.152643 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "daf312e60c014fe698d3ec9f1a1b769c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.153115 988 leader_election.cc:343] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.153224 988 leader_election.cc:304] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.153478 2374 raft_consensus.cc:2747] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.158413 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a2c4662f83d64bf896e58df3a6a5e140 (DEFAULT_TABLE table=default.Table_6d58bedddad3413d9c67e6aa0caaa6cb [id=11096ffbe13a4e5faa0864175d419d15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.158543 1155 tablet_bootstrap.cc:654] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.158855 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2c4662f83d64bf896e58df3a6a5e140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.171583 2404 tablet.cc:1722] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:51:57.173151 1155 tablet_bootstrap.cc:492] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.173511 1155 ts_tablet_manager.cc:1397] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250629 01:51:57.176942 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 74f3e7bf715c4032950ac095cb9b2e75 (DEFAULT_TABLE table=default.Table_b913ec7dcfc14a73ae38f883f9e5296d [id=ad32e1cae81849b795b21e0026a5cba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.176851 1155 raft_consensus.cc:357] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.177287 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74f3e7bf715c4032950ac095cb9b2e75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.177345 1155 raft_consensus.cc:383] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.177582 1155 raft_consensus.cc:738] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.178141 1155 consensus_queue.cc:260] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.179551 1155 ts_tablet_manager.cc:1428] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250629 01:51:57.182322 1155 tablet_bootstrap.cc:492] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.184201 1159 tablet_bootstrap.cc:492] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.184310 1159 ts_tablet_manager.cc:1397] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.025s
I20250629 01:51:57.184885 1159 raft_consensus.cc:357] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.185019 1159 raft_consensus.cc:383] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.185070 1159 raft_consensus.cc:738] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.185233 1159 consensus_queue.cc:260] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.185705 1159 ts_tablet_manager.cc:1428] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:57.185927 1159 tablet_bootstrap.cc:492] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.187965 1159 tablet_bootstrap.cc:654] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.189694 1155 tablet_bootstrap.cc:654] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.190384 1030 tablet_service.cc:1468] Processing CreateTablet for tablet aee0ea5e688c492885da7018e65d93e8 (DEFAULT_TABLE table=default.Table_5d856d8a80084e8bae95e734dff72636 [id=eebe2b05591547f3b58db56c4ade2216]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.190725 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aee0ea5e688c492885da7018e65d93e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.201185 753 tablet_service.cc:1468] Processing CreateTablet for tablet 29f6fa1b9ed44ca2b75a5b5d28127c5b (DEFAULT_TABLE table=default.Table_e5de441f126149e9810a9725612afbd7 [id=e2348b12711e4a03a9e49dc074de2782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.201472 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f6fa1b9ed44ca2b75a5b5d28127c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.202075 884 tablet_service.cc:1468] Processing CreateTablet for tablet 10477fa2ce6d4a03b743737a393dc575 (DEFAULT_TABLE table=default.Table_a6bd5d27120c4ca28bdd2f5ebc2170d1 [id=99d01016154e48fdbae972a9a1500fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.202629 881 tablet_service.cc:1468] Processing CreateTablet for tablet 027d08ece22d4b97a3c31e68bcaa396a (DEFAULT_TABLE table=default.Table_90112469ff66420a85ce68ec4cc69b56 [id=c0579faf605e416dbb709549c5f9d393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.202917 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027d08ece22d4b97a3c31e68bcaa396a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.203691 749 tablet_service.cc:1468] Processing CreateTablet for tablet c4c4153b6dcd45ada716c05d0590d10a (DEFAULT_TABLE table=default.Table_2f55e3c1a16e4082aaacb9434d6d583c [id=4718972c19d94c1bb83aee42b1768a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.203950 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c4153b6dcd45ada716c05d0590d10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.206382 747 tablet_service.cc:1468] Processing CreateTablet for tablet 10477fa2ce6d4a03b743737a393dc575 (DEFAULT_TABLE table=default.Table_a6bd5d27120c4ca28bdd2f5ebc2170d1 [id=99d01016154e48fdbae972a9a1500fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.206653 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10477fa2ce6d4a03b743737a393dc575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.207657 1159 tablet_bootstrap.cc:492] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.207798 1159 ts_tablet_manager.cc:1397] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20250629 01:51:57.208223 744 tablet_service.cc:1468] Processing CreateTablet for tablet 027d08ece22d4b97a3c31e68bcaa396a (DEFAULT_TABLE table=default.Table_90112469ff66420a85ce68ec4cc69b56 [id=c0579faf605e416dbb709549c5f9d393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.208494 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027d08ece22d4b97a3c31e68bcaa396a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.208756 1159 raft_consensus.cc:357] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.208946 1159 raft_consensus.cc:383] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.208941 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47cbc958eca14ed69fb45c3c2984aeef" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:57.209014 1159 raft_consensus.cc:738] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.209084 1049 raft_consensus.cc:2466] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:57.209228 1159 consensus_queue.cc:260] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.209764 1159 ts_tablet_manager.cc:1428] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:57.210044 2383 tablet.cc:1717] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Skipping requested alter to schema version 4, tablet already version 4
I20250629 01:51:57.210076 1159 tablet_bootstrap.cc:492] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.215268 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10477fa2ce6d4a03b743737a393dc575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.217996 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c1fb2548e78248e98b54bfb7f01c0e51" } modify_external_catalogs: true
I20250629 01:51:57.218204 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c1fb2548e78248e98b54bfb7f01c0e51" } modify_external_catalogs: true
I20250629 01:51:57.218773 881 tablet_service.cc:1468] Processing CreateTablet for tablet a78406b40f684843a08ae77732d12a8a (DEFAULT_TABLE table=default.Table_3911d6bceecf451cb29637a4843b711a [id=fc4388c084f04943a692754d7a6813a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.219108 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78406b40f684843a08ae77732d12a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.221647 1154 tablet_bootstrap.cc:492] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.221741 1154 ts_tablet_manager.cc:1397] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.226s user 0.036s sys 0.072s
I20250629 01:51:57.222283 1154 raft_consensus.cc:357] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.222409 1154 raft_consensus.cc:383] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.222452 1154 raft_consensus.cc:738] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.222596 1154 consensus_queue.cc:260] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.223003 1156 ts_tablet_manager.cc:1918] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.223182 1156 log.cc:1199] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e3b17172ab8b4e0dad36ebaac776dc03
I20250629 01:51:57.223544 1156 ts_tablet_manager.cc:1939] T e3b17172ab8b4e0dad36ebaac776dc03 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.224504 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1022) took 2402 ms. Trace:
I20250629 01:51:57.226145 1159 tablet_bootstrap.cc:654] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.232164 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b0ef1ba46ac487ab1502c3d116814e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.232520 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ff26f8ed91be432fb708a9dad260b30d (DEFAULT_TABLE table=default.Table_54c1b81ff73b436fbf764565c0ab55ca [id=f55f6cf2078e42dd843fb07a85afcc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.232837 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff26f8ed91be432fb708a9dad260b30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.236374 1154 ts_tablet_manager.cc:1428] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250629 01:51:57.236611 1154 tablet_bootstrap.cc:492] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.237823 1154 tablet_bootstrap.cc:654] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.238101 1157 ts_tablet_manager.cc:1939] T 20958c5e88454e129aa8efefb63b7f14 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:57.238655 1157 tablet_replica.cc:331] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:57.238914 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]) successfully deleted
W20250629 01:51:57.239173 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 33aa0a66cdbf49c5820e2b43b021f28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33aa0a66cdbf49c5820e2b43b021f28b already in progress: creating tablet
I20250629 01:51:57.239275 1156 rpcz_store.cc:276] 0629 01:51:54.822326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.822367 (+ 41us) service_pool.cc:225] Handling call
0629 01:51:57.048633 (+2226266us) ts_tablet_manager.cc:1145] Deleting tablet e3b17172ab8b4e0dad36ebaac776dc03
0629 01:51:57.048640 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.051704 (+ 3064us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.222924 (+171220us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.224489 (+ 1565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226114}
I20250629 01:51:57.240274 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e3b17172ab8b4e0dad36ebaac776dc03 (table default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]) successfully deleted
I20250629 01:51:57.240571 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1023) took 2406 ms. Trace:
I20250629 01:51:57.240721 1157 raft_consensus.cc:2241] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.240885 1157 raft_consensus.cc:2270] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.240971 1156 rpcz_store.cc:276] 0629 01:51:54.833618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.834275 (+ 657us) service_pool.cc:225] Handling call
0629 01:51:57.240474 (+2406199us) ts_tablet_manager.cc:1145] Deleting tablet 5d2f2975e5d64efd888ed948f1420916
0629 01:51:57.240482 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.240556 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406077}
W20250629 01:51:57.241452 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5d2f2975e5d64efd888ed948f1420916 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d2f2975e5d64efd888ed948f1420916 already in progress: creating tablet
I20250629 01:51:57.241580 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1026) took 2357 ms. Trace:
I20250629 01:51:57.241672 1156 rpcz_store.cc:276] 0629 01:51:54.884380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.884514 (+ 134us) service_pool.cc:225] Handling call
0629 01:51:57.241441 (+2356927us) ts_tablet_manager.cc:1145] Deleting tablet e332a512574e4b4e881aeb37e289f0bc
0629 01:51:57.241450 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.241565 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2356789}
I20250629 01:51:57.241918 1157 ts_tablet_manager.cc:1905] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:57.241992 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e332a512574e4b4e881aeb37e289f0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e332a512574e4b4e881aeb37e289f0bc already in progress: creating tablet
I20250629 01:51:57.242174 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1027) took 2353 ms. Trace:
I20250629 01:51:57.242264 1156 rpcz_store.cc:276] 0629 01:51:54.889060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.889129 (+ 69us) service_pool.cc:225] Handling call
0629 01:51:57.241989 (+2352860us) ts_tablet_manager.cc:1145] Deleting tablet a53d9e8b7efe46099821ae628640a42e
0629 01:51:57.242040 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.242161 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352067}
W20250629 01:51:57.242624 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a53d9e8b7efe46099821ae628640a42e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a53d9e8b7efe46099821ae628640a42e already in progress: creating tablet
I20250629 01:51:57.244480 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:57.242671 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1025) took 2359 ms. Trace:
I20250629 01:51:57.244691 1156 rpcz_store.cc:276] 0629 01:51:54.883500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.889987 (+ 6487us) service_pool.cc:225] Handling call
0629 01:51:57.242592 (+2352605us) ts_tablet_manager.cc:1145] Deleting tablet 60e12d28ef784d898dbf9846f7063fdf
0629 01:51:57.242600 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.242659 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352411}
W20250629 01:51:57.245513 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 60e12d28ef784d898dbf9846f7063fdf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60e12d28ef784d898dbf9846f7063fdf already in progress: creating tablet
I20250629 01:51:57.264925 1158 ts_tablet_manager.cc:1918] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.265067 1158 log.cc:1199] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/0d4320f615464fedbcab25137952f684
I20250629 01:51:57.265394 1158 ts_tablet_manager.cc:1939] T 0d4320f615464fedbcab25137952f684 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:57.267356 1157 ts_tablet_manager.cc:1918] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.267724 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1028) took 2375 ms. Trace:
I20250629 01:51:57.267910 1156 rpcz_store.cc:276] 0629 01:51:54.891770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:54.898816 (+ 7046us) service_pool.cc:225] Handling call
0629 01:51:57.245035 (+2346219us) ts_tablet_manager.cc:1145] Deleting tablet e5d27d658892496a9b19b57949a8399a
0629 01:51:57.267579 (+ 22544us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.267707 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346053}
I20250629 01:51:57.267846 1157 log.cc:1199] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/cabb0298771e4fd5be29e530d3a8ed3d
I20250629 01:51:57.269088 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1039) took 2255 ms. Trace:
W20250629 01:51:57.269555 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e5d27d658892496a9b19b57949a8399a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5d27d658892496a9b19b57949a8399a already in progress: creating tablet
I20250629 01:51:57.270181 1157 ts_tablet_manager.cc:1939] T cabb0298771e4fd5be29e530d3a8ed3d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:57.271696 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet cabb0298771e4fd5be29e530d3a8ed3d (table default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]) successfully deleted
W20250629 01:51:57.272958 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6b0ef1ba46ac487ab1502c3d116814e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b0ef1ba46ac487ab1502c3d116814e1 already in progress: creating tablet
W20250629 01:51:57.274176 442 catalog_manager.cc:4895] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 962f1ac223074f11a69061a69396b3aa because the tablet was not found. No further retry: Not found: Tablet not found: 962f1ac223074f11a69061a69396b3aa
I20250629 01:51:57.275110 1157 tablet_replica.cc:331] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:57.275974 1157 raft_consensus.cc:2241] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.276602 1157 raft_consensus.cc:2270] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.279016 1157 ts_tablet_manager.cc:1905] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.279515 1156 rpcz_store.cc:276] 0629 01:51:55.013613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.013707 (+ 94us) service_pool.cc:225] Handling call
0629 01:51:57.268977 (+2255270us) ts_tablet_manager.cc:1145] Deleting tablet ac560295fddd44368e6f9bc543f7cbfa
0629 01:51:57.268987 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.269073 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2255110}
I20250629 01:51:57.280552 1156 tablet_replica.cc:331] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.281260 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:57.283335 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_803f2f1324164400bb38186cbb242bcb" } modify_external_catalogs: true
I20250629 01:51:57.283633 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_803f2f1324164400bb38186cbb242bcb" } modify_external_catalogs: true
W20250629 01:51:57.287307 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 33aa0a66cdbf49c5820e2b43b021f28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33aa0a66cdbf49c5820e2b43b021f28b already in progress: creating tablet
I20250629 01:51:57.287693 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet cb7314fbdfc74b1c8d9d1be9c05244cb (table default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]) successfully deleted
I20250629 01:51:57.280814 2337 raft_consensus.cc:491] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.289565 2337 raft_consensus.cc:513] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.289680 890 tablet_service.cc:1468] Processing CreateTablet for tablet 962f1ac223074f11a69061a69396b3aa (DEFAULT_TABLE table=default.Table_d5e5c0a7d4814ee6895122e2eb2a0717 [id=42eb08c47dc146f5b972799df312a610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.289945 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962f1ac223074f11a69061a69396b3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.290098 2337 leader_election.cc:290] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:57.290231 877 tablet_service.cc:1468] Processing CreateTablet for tablet 29f6fa1b9ed44ca2b75a5b5d28127c5b (DEFAULT_TABLE table=default.Table_e5de441f126149e9810a9725612afbd7 [id=e2348b12711e4a03a9e49dc074de2782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.290472 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f6fa1b9ed44ca2b75a5b5d28127c5b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:57.290859 710 leader_election.cc:343] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a6a012ba891949538800581436ef5380
I20250629 01:51:57.290971 710 leader_election.cc:304] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:57.291229 2332 raft_consensus.cc:2747] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.291296 710 leader_election.cc:304] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.291474 2332 raft_consensus.cc:2802] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:57.291530 2332 raft_consensus.cc:491] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:57.291581 2332 raft_consensus.cc:3058] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.292754 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 0d4320f615464fedbcab25137952f684 (table default.Table_c7d1c2fe02d04423803f16f69f0d6625 [id=5a75eb0020fb4d4fb1d2066ed7d670a5]) successfully deleted
W20250629 01:51:57.292958 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 962f1ac223074f11a69061a69396b3aa because the tablet was not found. No further retry: Not found: Tablet not found: 962f1ac223074f11a69061a69396b3aa
I20250629 01:51:57.289283 744 tablet_service.cc:1468] Processing CreateTablet for tablet a78406b40f684843a08ae77732d12a8a (DEFAULT_TABLE table=default.Table_3911d6bceecf451cb29637a4843b711a [id=fc4388c084f04943a692754d7a6813a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.293810 2332 raft_consensus.cc:513] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.294656 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47cbc958eca14ed69fb45c3c2984aeef" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:57.294219 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78406b40f684843a08ae77732d12a8a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:57.295090 712 leader_election.cc:343] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.294250 2332 leader_election.cc:290] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.298633 753 tablet_service.cc:1468] Processing CreateTablet for tablet 962f1ac223074f11a69061a69396b3aa (DEFAULT_TABLE table=default.Table_d5e5c0a7d4814ee6895122e2eb2a0717 [id=42eb08c47dc146f5b972799df312a610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.298949 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962f1ac223074f11a69061a69396b3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.299037 2363 raft_consensus.cc:491] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.299127 2363 raft_consensus.cc:513] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.299562 2363 leader_election.cc:290] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.300005 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82a837932f14419c8b32fcd2c9358215" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.300372 988 leader_election.cc:343] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.300722 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82a837932f14419c8b32fcd2c9358215" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:57.300843 775 raft_consensus.cc:2466] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:57.301136 986 leader_election.cc:304] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.301311 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347fa6a12a744ee88058643b7143e39d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.301446 2363 raft_consensus.cc:2802] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:57.301505 2363 raft_consensus.cc:491] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:57.301556 2363 raft_consensus.cc:3058] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:57.301693 712 leader_election.cc:343] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.303715 2363 raft_consensus.cc:513] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.304445 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82a837932f14419c8b32fcd2c9358215" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:57.304571 775 raft_consensus.cc:3058] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:57.305445 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6b0ef1ba46ac487ab1502c3d116814e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b0ef1ba46ac487ab1502c3d116814e1 already in progress: creating tablet
I20250629 01:51:57.306653 775 raft_consensus.cc:2466] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:57.307541 1157 ts_tablet_manager.cc:1918] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.307957 986 leader_election.cc:304] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:57.307941 1157 log.cc:1199] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4e354fce951943d69e4c666897f5caac
I20250629 01:51:57.308351 2363 leader_election.cc:290] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.308482 2363 raft_consensus.cc:2802] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:57.308723 2363 raft_consensus.cc:695] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:57.308972 2363 consensus_queue.cc:237] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.309783 1157 ts_tablet_manager.cc:1939] T 4e354fce951943d69e4c666897f5caac P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:57.309881 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82a837932f14419c8b32fcd2c9358215" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:57.310463 988 leader_election.cc:343] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.310993 1156 raft_consensus.cc:2241] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.311841 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4e354fce951943d69e4c666897f5caac (table default.Table_67f686028f0c40c991d3231bd0b0c823 [id=c51c68dd349543d4a4a8bfb1b9aa1184]) successfully deleted
I20250629 01:51:57.313380 1156 raft_consensus.cc:2270] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.316628 1156 ts_tablet_manager.cc:1905] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.317070 881 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d21dfd48ea4a95a2fcecde6735974b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.317631 1157 tablet_replica.cc:331] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:57.317886 1157 raft_consensus.cc:2241] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.318075 1157 raft_consensus.cc:2270] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.318993 1157 ts_tablet_manager.cc:1905] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.319360 2411 tablet.cc:1717] T 0323947b135a4308903b1eed1ca4db71 P 443c4bd3c5d548ee8cd347d2b4de5293: Skipping requested alter to schema version 3, tablet already version 3
I20250629 01:51:57.320334 2049 consensus_queue.cc:1035] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:57.322432 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c4c4153b6dcd45ada716c05d0590d10a (DEFAULT_TABLE table=default.Table_2f55e3c1a16e4082aaacb9434d6d583c [id=4718972c19d94c1bb83aee42b1768a9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.322757 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c4153b6dcd45ada716c05d0590d10a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.326020 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 29f6fa1b9ed44ca2b75a5b5d28127c5b (DEFAULT_TABLE table=default.Table_e5de441f126149e9810a9725612afbd7 [id=e2348b12711e4a03a9e49dc074de2782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.326301 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29f6fa1b9ed44ca2b75a5b5d28127c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.330085 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_904b8b7a3ab1414a9bd29274648bbb86 [id=e991e29b3d9d4dcc96912754565336f8] alter complete (version 1)
I20250629 01:51:57.331137 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6f92518779264d63a2ed0d8468186644"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.331588 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f92518779264d63a2ed0d8468186644 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.331703 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 10477fa2ce6d4a03b743737a393dc575 (DEFAULT_TABLE table=default.Table_a6bd5d27120c4ca28bdd2f5ebc2170d1 [id=99d01016154e48fdbae972a9a1500fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.331992 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10477fa2ce6d4a03b743737a393dc575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.340456 2412 raft_consensus.cc:491] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.340615 2412 raft_consensus.cc:513] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.340739 1157 ts_tablet_manager.cc:1918] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.340862 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.341241 1157 log.cc:1199] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b6d21dfd48ea4a95a2fcecde6735974b
I20250629 01:51:57.341915 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20586ba2af9e4cb7adec5ca6c3b76e42" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.342319 712 leader_election.cc:343] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.342489 2412 leader_election.cc:290] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.342592 1157 ts_tablet_manager.cc:1939] T b6d21dfd48ea4a95a2fcecde6735974b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:57.344132 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b6d21dfd48ea4a95a2fcecde6735974b (table default.Table_1b37a535cd2f483a8c72aa10882a4301 [id=b516da4148384818bcc69e4259f20ef9]) successfully deleted
W20250629 01:51:57.346038 712 consensus_peers.cc:489] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:57.346858 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a78406b40f684843a08ae77732d12a8a (DEFAULT_TABLE table=default.Table_3911d6bceecf451cb29637a4843b711a [id=fc4388c084f04943a692754d7a6813a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.347179 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a78406b40f684843a08ae77732d12a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.347787 1156 ts_tablet_manager.cc:1918] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.352174 1156 log.cc:1199] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/da93c730451d42bc8df1e582fb3af818
I20250629 01:51:57.352720 1156 ts_tablet_manager.cc:1939] T da93c730451d42bc8df1e582fb3af818 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.353037 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1040) took 2224 ms. Trace:
I20250629 01:51:57.353153 1156 rpcz_store.cc:276] 0629 01:51:55.128213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.128260 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:57.280507 (+2152247us) ts_tablet_manager.cc:1145] Deleting tablet da93c730451d42bc8df1e582fb3af818
0629 01:51:57.280517 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.324419 (+ 43902us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.347710 (+ 23291us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.353021 (+ 5311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2152090}
I20250629 01:51:57.353659 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 027d08ece22d4b97a3c31e68bcaa396a (DEFAULT_TABLE table=default.Table_90112469ff66420a85ce68ec4cc69b56 [id=c0579faf605e416dbb709549c5f9d393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.353900 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 027d08ece22d4b97a3c31e68bcaa396a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.355888 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet da93c730451d42bc8df1e582fb3af818 (table default.Table_2e7ef5453b1945728384798f25e47b7f [id=88472d60f4244733beb9ff0fd6ce9c9f]) successfully deleted
I20250629 01:51:57.356215 1156 tablet_replica.cc:331] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.356457 1156 raft_consensus.cc:2241] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.357898 2049 raft_consensus.cc:491] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.358971 2049 raft_consensus.cc:513] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.359405 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 2249d5cba1544653b088312e7d74802e with delete_type TABLET_DATA_DELETED (replaced by tablet d387af53209147a6bac7afeda9a411ab) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.359506 2049 leader_election.cc:290] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.360046 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.360373 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 2249d5cba1544653b088312e7d74802e with delete_type TABLET_DATA_DELETED (replaced by tablet d387af53209147a6bac7afeda9a411ab) from {username='slave'} at 127.0.0.1:57444
W20250629 01:51:57.360507 712 leader_election.cc:343] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.360525 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet daf312e60c014fe698d3ec9f1a1b769c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.360775 1158 tablet_replica.cc:331] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.360807 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cfff25bea764e62954e49611b71703c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.361006 1158 raft_consensus.cc:2241] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:57.361212 1158 pending_rounds.cc:70] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:57.361250 1158 pending_rounds.cc:77] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759211778994176 op_type: NO_OP noop_request { }
I20250629 01:51:57.361326 1158 raft_consensus.cc:2887] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:57.361394 1158 raft_consensus.cc:2270] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.362057 1158 ts_tablet_manager.cc:1905] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.362116 1156 raft_consensus.cc:2270] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.362273 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "47cbc958eca14ed69fb45c3c2984aeef" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:57.362406 1050 raft_consensus.cc:3058] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.362584 1156 ts_tablet_manager.cc:1905] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.364481 1050 raft_consensus.cc:2466] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:57.367457 1158 ts_tablet_manager.cc:1918] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:57.367612 1158 log.cc:1199] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/2249d5cba1544653b088312e7d74802e
I20250629 01:51:57.368016 1158 ts_tablet_manager.cc:1939] T 2249d5cba1544653b088312e7d74802e P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:57.368752 1158 tablet_replica.cc:331] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.379570 1155 tablet_bootstrap.cc:492] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.379973 1155 ts_tablet_manager.cc:1397] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.198s user 0.007s sys 0.050s
I20250629 01:51:57.381685 710 leader_election.cc:304] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.381886 1158 raft_consensus.cc:2241] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.381997 1158 raft_consensus.cc:2270] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.382047 2049 raft_consensus.cc:2802] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:57.382166 2049 raft_consensus.cc:695] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:57.382406 2049 consensus_queue.cc:237] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.382881 1155 raft_consensus.cc:357] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.383396 1155 raft_consensus.cc:383] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.383603 1155 raft_consensus.cc:738] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.384152 1155 consensus_queue.cc:260] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.384763 1156 ts_tablet_manager.cc:1918] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.384990 1156 log.cc:1199] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/2aca26af31ed4bc994f4d62603edc881
I20250629 01:51:57.384970 1158 ts_tablet_manager.cc:1905] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.385311 1156 ts_tablet_manager.cc:1939] T 2aca26af31ed4bc994f4d62603edc881 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.385964 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1042) took 2239 ms. Trace:
I20250629 01:51:57.385991 1155 ts_tablet_manager.cc:1428] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:57.370215 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347fa6a12a744ee88058643b7143e39d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:57.388342 1156 rpcz_store.cc:276] 0629 01:51:55.146664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.146707 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:57.356166 (+2209459us) ts_tablet_manager.cc:1145] Deleting tablet 2aca26af31ed4bc994f4d62603edc881
0629 01:51:57.356176 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.365505 (+ 9329us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.382634 (+ 17129us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.385947 (+ 3313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2209310}
I20250629 01:51:57.388494 1048 raft_consensus.cc:2466] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:57.388716 1155 tablet_bootstrap.cc:492] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.388782 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 2aca26af31ed4bc994f4d62603edc881 (table default.Table_8bf46b174da647d2b1135f050ce5e430 [id=72ebf1394e9a40faaee64e1b26e6ffe6]) successfully deleted
I20250629 01:51:57.390360 1154 tablet_bootstrap.cc:492] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.390533 1154 ts_tablet_manager.cc:1397] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.154s user 0.015s sys 0.019s
I20250629 01:51:57.390596 1156 tablet_replica.cc:331] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.390818 1156 raft_consensus.cc:2241] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.390916 1155 tablet_bootstrap.cc:654] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.391003 1156 raft_consensus.cc:2270] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.392212 1156 ts_tablet_manager.cc:1905] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.393682 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 2249d5cba1544653b088312e7d74802e (table default.Table_c1fb2548e78248e98b54bfb7f01c0e51 [id=a2b08c6d970a45709f099af55d84f522]) successfully deleted
I20250629 01:51:57.394104 1159 tablet_bootstrap.cc:492] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.378836 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 258994dd24804f4aaa5e299436fac9b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.394281 1159 ts_tablet_manager.cc:1397] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.184s user 0.007s sys 0.047s
I20250629 01:51:57.380080 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 962f1ac223074f11a69061a69396b3aa (DEFAULT_TABLE table=default.Table_d5e5c0a7d4814ee6895122e2eb2a0717 [id=42eb08c47dc146f5b972799df312a610]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.381443 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:57.394475 1052 raft_consensus.cc:2466] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:57.395043 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 962f1ac223074f11a69061a69396b3aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.395279 1159 raft_consensus.cc:357] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.395637 1159 raft_consensus.cc:383] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.395809 2371 tablet.cc:1722] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:57.395808 1154 raft_consensus.cc:357] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.395941 1154 raft_consensus.cc:383] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.395987 1154 raft_consensus.cc:738] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:51:57.395984 710 leader_election.cc:343] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 20586ba2af9e4cb7adec5ca6c3b76e42
I20250629 01:51:57.396070 710 leader_election.cc:304] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:57.396174 1154 consensus_queue.cc:260] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.396384 2337 raft_consensus.cc:2747] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.397888 1154 ts_tablet_manager.cc:1428] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250629 01:51:57.397902 2217 raft_consensus.cc:491] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.398005 2217 raft_consensus.cc:513] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.398520 2217 leader_election.cc:290] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:57.399330 1159 raft_consensus.cc:738] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.399607 1159 consensus_queue.cc:260] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.399829 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:57.401274 1154 tablet_bootstrap.cc:492] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.402534 1156 ts_tablet_manager.cc:1918] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.402678 1156 log.cc:1199] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/1c995ce14c9a4966b842fcba520cf87d
I20250629 01:51:57.402995 1156 ts_tablet_manager.cc:1939] T 1c995ce14c9a4966b842fcba520cf87d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.403708 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1041) took 2257 ms. Trace:
I20250629 01:51:57.403832 1156 rpcz_store.cc:276] 0629 01:51:55.146563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.146882 (+ 319us) service_pool.cc:225] Handling call
0629 01:51:57.390558 (+2243676us) ts_tablet_manager.cc:1145] Deleting tablet 1c995ce14c9a4966b842fcba520cf87d
0629 01:51:57.390567 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.394798 (+ 4231us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.402463 (+ 7665us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.403690 (+ 1227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2243482}
I20250629 01:51:57.399487 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ddc24d0e668c49ac825b2578849ab2a3" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:57.404467 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 1c995ce14c9a4966b842fcba520cf87d (table default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]) successfully deleted
I20250629 01:51:57.404516 850 leader_election.cc:304] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.404801 2217 raft_consensus.cc:2747] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.404922 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1043) took 2254 ms. Trace:
I20250629 01:51:57.405022 1156 rpcz_store.cc:276] 0629 01:51:55.150494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.150662 (+ 168us) service_pool.cc:225] Handling call
0629 01:51:57.404740 (+2254078us) ts_tablet_manager.cc:1145] Deleting tablet ee1baa976b5e47e490a0fd561b37f83d
0629 01:51:57.404749 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.404907 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2253934}
I20250629 01:51:57.405404 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
W20250629 01:51:57.405479 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ee1baa976b5e47e490a0fd561b37f83d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee1baa976b5e47e490a0fd561b37f83d already in progress: creating tablet
I20250629 01:51:57.405715 1158 ts_tablet_manager.cc:1918] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.405786 1158 log.cc:1199] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:51:57.405797 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1044) took 2255 ms. Trace:
I20250629 01:51:57.405866 1156 rpcz_store.cc:276] 0629 01:51:55.150618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.150829 (+ 211us) service_pool.cc:225] Handling call
0629 01:51:57.405696 (+2254867us) ts_tablet_manager.cc:1145] Deleting tablet caa63fe10ff9487a9fc16479285c5abd
0629 01:51:57.405706 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.405781 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254707}
I20250629 01:51:57.406016 1158 ts_tablet_manager.cc:1939] T daf312e60c014fe698d3ec9f1a1b769c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:51:57.406239 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet caa63fe10ff9487a9fc16479285c5abd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caa63fe10ff9487a9fc16479285c5abd already in progress: creating tablet
I20250629 01:51:57.406541 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1045) took 2254 ms. Trace:
I20250629 01:51:57.406705 1154 tablet_bootstrap.cc:654] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.406610 1156 rpcz_store.cc:276] 0629 01:51:55.151627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.151671 (+ 44us) service_pool.cc:225] Handling call
0629 01:51:57.406444 (+2254773us) ts_tablet_manager.cc:1145] Deleting tablet 8cea647451954d8ab3bd495a3a080c97
0629 01:51:57.406453 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.406525 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2254635}
W20250629 01:51:57.406987 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8cea647451954d8ab3bd495a3a080c97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8cea647451954d8ab3bd495a3a080c97 already in progress: creating tablet
I20250629 01:51:57.407286 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1046) took 2252 ms. Trace:
I20250629 01:51:57.407351 1156 rpcz_store.cc:276] 0629 01:51:55.154371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.154410 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:57.407190 (+2252780us) ts_tablet_manager.cc:1145] Deleting tablet 06d591c66d734f76a1bda2194ec62776
0629 01:51:57.407200 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.407271 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252611}
W20250629 01:51:57.407716 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 06d591c66d734f76a1bda2194ec62776 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06d591c66d734f76a1bda2194ec62776 already in progress: creating tablet
I20250629 01:51:57.407999 1156 tablet_replica.cc:331] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.408244 1156 raft_consensus.cc:2241] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.408423 1156 raft_consensus.cc:2270] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.409022 1156 ts_tablet_manager.cc:1905] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.410054 1154 tablet_bootstrap.cc:492] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.410518 1154 ts_tablet_manager.cc:1397] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250629 01:51:57.411564 1154 raft_consensus.cc:357] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.411638 1159 ts_tablet_manager.cc:1428] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20250629 01:51:57.413990 710 leader_election.cc:304] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.414274 1156 ts_tablet_manager.cc:1918] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.414399 1156 log.cc:1199] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/52be184d0480495a894442d6f72a0f12
I20250629 01:51:57.414424 710 leader_election.cc:304] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.414669 1156 ts_tablet_manager.cc:1939] T 52be184d0480495a894442d6f72a0f12 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.414695 2337 raft_consensus.cc:2802] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:57.414746 2337 raft_consensus.cc:491] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:57.414808 2337 raft_consensus.cc:3058] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.415167 1159 tablet_bootstrap.cc:492] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.415220 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1059) took 2100 ms. Trace:
I20250629 01:51:57.415277 1156 rpcz_store.cc:276] 0629 01:51:55.315045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.318169 (+ 3124us) service_pool.cc:225] Handling call
0629 01:51:57.407957 (+2089788us) ts_tablet_manager.cc:1145] Deleting tablet 52be184d0480495a894442d6f72a0f12
0629 01:51:57.407967 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.411888 (+ 3921us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.414214 (+ 2326us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.415204 (+ 990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2089617}
I20250629 01:51:57.415607 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 52be184d0480495a894442d6f72a0f12 (table default.Table_cbbc85d10dc546b59c06dd6296eaa2db [id=fadb6eabbead48b1a11795ffed0c4d81]) successfully deleted
I20250629 01:51:57.415890 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1080) took 1861 ms. Trace:
I20250629 01:51:57.415951 1156 rpcz_store.cc:276] 0629 01:51:55.554214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.554254 (+ 40us) service_pool.cc:225] Handling call
0629 01:51:57.415798 (+1861544us) ts_tablet_manager.cc:1145] Deleting tablet 9ed1ee73bb8f4761806a66f988b57bf9
0629 01:51:57.415805 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.415868 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861409}
W20250629 01:51:57.416311 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9ed1ee73bb8f4761806a66f988b57bf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ed1ee73bb8f4761806a66f988b57bf9 already in progress: creating tablet
I20250629 01:51:57.416608 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1081) took 1859 ms. Trace:
I20250629 01:51:57.416667 1156 rpcz_store.cc:276] 0629 01:51:55.557126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.557165 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:57.416513 (+1859348us) ts_tablet_manager.cc:1145] Deleting tablet 303f77a729c648f9b1146d11056b242d
0629 01:51:57.416522 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.416593 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859228}
I20250629 01:51:57.416936 2337 raft_consensus.cc:513] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:57.417063 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 303f77a729c648f9b1146d11056b242d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 303f77a729c648f9b1146d11056b242d already in progress: creating tablet
I20250629 01:51:57.417330 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1082) took 1859 ms. Trace:
I20250629 01:51:57.417419 2337 leader_election.cc:290] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:57.417580 2337 raft_consensus.cc:2802] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:57.417639 2337 raft_consensus.cc:491] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:57.417678 2337 raft_consensus.cc:3058] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.417662 1159 tablet_bootstrap.cc:654] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.417764 1154 raft_consensus.cc:383] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.417815 1154 raft_consensus.cc:738] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.418164 1154 consensus_queue.cc:260] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.418973 1154 ts_tablet_manager.cc:1428] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250629 01:51:57.419095 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347fa6a12a744ee88058643b7143e39d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:57.419198 1154 tablet_bootstrap.cc:492] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:57.419636 712 leader_election.cc:343] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.420426 1154 tablet_bootstrap.cc:654] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.421553 2337 raft_consensus.cc:513] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.422034 1155 tablet_bootstrap.cc:492] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.422209 2337 leader_election.cc:290] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.422602 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:57.423084 712 leader_election.cc:343] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.423274 1155 ts_tablet_manager.cc:1397] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.006s
I20250629 01:51:57.420714 1158 tablet_replica.cc:331] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.424063 1155 raft_consensus.cc:357] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.424222 1155 raft_consensus.cc:383] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.424288 1155 raft_consensus.cc:738] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.424507 1158 raft_consensus.cc:2241] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.424455 1155 consensus_queue.cc:260] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.424690 1158 raft_consensus.cc:2270] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.425061 1155 ts_tablet_manager.cc:1428] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:57.425282 1155 tablet_bootstrap.cc:492] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.425310 1158 ts_tablet_manager.cc:1905] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.417389 1156 rpcz_store.cc:276] 0629 01:51:55.557934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.557975 (+ 41us) service_pool.cc:225] Handling call
0629 01:51:57.417247 (+1859272us) ts_tablet_manager.cc:1145] Deleting tablet 3ba5887dffc1460a89208447ecc6a198
0629 01:51:57.417255 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.417316 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859165}
W20250629 01:51:57.426733 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 3ba5887dffc1460a89208447ecc6a198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ba5887dffc1460a89208447ecc6a198 already in progress: creating tablet
I20250629 01:51:57.427089 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1083) took 1867 ms. Trace:
I20250629 01:51:57.427172 1156 rpcz_store.cc:276] 0629 01:51:55.559178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.559226 (+ 48us) service_pool.cc:225] Handling call
0629 01:51:57.426979 (+1867753us) ts_tablet_manager.cc:1145] Deleting tablet 2521a476643c41c1a2e5282755a58091
0629 01:51:57.426989 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.427071 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1867627}
W20250629 01:51:57.427567 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2521a476643c41c1a2e5282755a58091 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2521a476643c41c1a2e5282755a58091 already in progress: creating tablet
I20250629 01:51:57.428309 1155 tablet_bootstrap.cc:654] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.428697 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet daf312e60c014fe698d3ec9f1a1b769c (table default.Table_3f1b07c24c6441f4a4965f8e055648fe [id=df4a1725208848efa85ea19a1d3da531]) successfully deleted
I20250629 01:51:57.429005 1159 tablet_bootstrap.cc:492] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.429188 1159 ts_tablet_manager.cc:1397] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20250629 01:51:57.429271 2383 tablet.cc:1722] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:57.429776 2418 raft_consensus.cc:491] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.429891 2418 raft_consensus.cc:513] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.430358 2418 leader_election.cc:290] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:57.430816 1159 raft_consensus.cc:357] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.431139 1159 raft_consensus.cc:383] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.431245 1159 raft_consensus.cc:738] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.431465 1159 consensus_queue.cc:260] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:57.432543 850 leader_election.cc:343] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 6f3a65c9c1c740b1865b96a6e2aea524
I20250629 01:51:57.432925 1154 tablet_bootstrap.cc:492] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.434124 1159 ts_tablet_manager.cc:1428] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:57.434432 1159 tablet_bootstrap.cc:492] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.433655 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "347fa6a12a744ee88058643b7143e39d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:57.434710 1052 raft_consensus.cc:3058] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:57.435475 850 leader_election.cc:343] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 6f3a65c9c1c740b1865b96a6e2aea524
I20250629 01:51:57.435559 850 leader_election.cc:304] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.435523 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1084) took 1874 ms. Trace:
I20250629 01:51:57.435930 2418 raft_consensus.cc:2747] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.435670 1156 rpcz_store.cc:276] 0629 01:51:55.561061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.561194 (+ 133us) service_pool.cc:225] Handling call
0629 01:51:57.428288 (+1867094us) ts_tablet_manager.cc:1145] Deleting tablet a2e9de48e4ad401880f351f2bd55cf5c
0629 01:51:57.428295 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.435427 (+ 7132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866963}
I20250629 01:51:57.436169 1154 ts_tablet_manager.cc:1397] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
W20250629 01:51:57.436322 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a2e9de48e4ad401880f351f2bd55cf5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e9de48e4ad401880f351f2bd55cf5c already in progress: creating tablet
I20250629 01:51:57.436462 1156 tablet_replica.cc:331] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.437129 1156 raft_consensus.cc:2241] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.437662 1156 raft_consensus.cc:2270] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.437012 1052 raft_consensus.cc:2466] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:57.434247 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:57.438504 1048 raft_consensus.cc:3058] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.440457 1155 tablet_bootstrap.cc:492] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.440598 1155 ts_tablet_manager.cc:1397] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20250629 01:51:57.441208 1155 raft_consensus.cc:357] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.441388 1155 raft_consensus.cc:383] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.441483 1155 raft_consensus.cc:738] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.441599 1159 tablet_bootstrap.cc:654] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.441677 1155 consensus_queue.cc:260] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.442147 1155 ts_tablet_manager.cc:1428] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:51:57.442284 1156 ts_tablet_manager.cc:1905] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.442440 1155 tablet_bootstrap.cc:492] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.442708 1154 raft_consensus.cc:357] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.442880 1154 raft_consensus.cc:383] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.442953 1154 raft_consensus.cc:738] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.444245 1155 tablet_bootstrap.cc:654] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.444989 1154 consensus_queue.cc:260] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.445508 1154 ts_tablet_manager.cc:1428] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250629 01:51:57.445744 1154 tablet_bootstrap.cc:492] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.446979 1154 tablet_bootstrap.cc:654] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.448645 2336 raft_consensus.cc:491] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.448758 2336 raft_consensus.cc:513] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.449239 2336 leader_election.cc:290] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.450073 916 raft_consensus.cc:3058] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.452973 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4320f615464fedbcab25137952f684" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.448483 1158 ts_tablet_manager.cc:1918] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.453377 1158 log.cc:1199] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9cfff25bea764e62954e49611b71703c
I20250629 01:51:57.453613 1158 ts_tablet_manager.cc:1939] T 9cfff25bea764e62954e49611b71703c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:57.453907 1158 tablet_replica.cc:331] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.455390 1156 ts_tablet_manager.cc:1918] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:57.455684 712 leader_election.cc:343] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.455765 1156 log.cc:1199] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/c0b9afb9976541b2bc7ab188f35a62ec
I20250629 01:51:57.456228 1158 raft_consensus.cc:2241] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.456347 1158 raft_consensus.cc:2270] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.456701 1158 ts_tablet_manager.cc:1905] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.456902 1156 ts_tablet_manager.cc:1939] T c0b9afb9976541b2bc7ab188f35a62ec P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.451669 1048 raft_consensus.cc:2466] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:57.460139 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1086) took 1725 ms. Trace:
I20250629 01:51:57.462087 1156 rpcz_store.cc:276] 0629 01:51:55.734047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.734092 (+ 45us) service_pool.cc:225] Handling call
0629 01:51:57.436328 (+1702236us) ts_tablet_manager.cc:1145] Deleting tablet c0b9afb9976541b2bc7ab188f35a62ec
0629 01:51:57.436339 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.447379 (+ 11040us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.455140 (+ 7761us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.459979 (+ 4839us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702004}
I20250629 01:51:57.463141 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet c0b9afb9976541b2bc7ab188f35a62ec (table default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]) successfully deleted
I20250629 01:51:57.463680 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1095) took 1682 ms. Trace:
I20250629 01:51:57.463999 1156 rpcz_store.cc:276] 0629 01:51:55.781142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.781314 (+ 172us) service_pool.cc:225] Handling call
0629 01:51:57.463183 (+1681869us) ts_tablet_manager.cc:1145] Deleting tablet 11b4bdbbf5d3421ab6e3a31de90fff6d
0629 01:51:57.463246 (+ 63us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.463566 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1681732}
W20250629 01:51:57.464969 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 11b4bdbbf5d3421ab6e3a31de90fff6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11b4bdbbf5d3421ab6e3a31de90fff6d already in progress: creating tablet
I20250629 01:51:57.465355 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1096) took 1684 ms. Trace:
I20250629 01:51:57.466089 2337 consensus_queue.cc:1035] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:57.465685 1156 rpcz_store.cc:276] 0629 01:51:55.781236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.781470 (+ 234us) service_pool.cc:225] Handling call
0629 01:51:57.464965 (+1683495us) ts_tablet_manager.cc:1145] Deleting tablet 556c2050d2cb40f28a3a0bfb9d677d69
0629 01:51:57.465027 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.465293 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683320}
W20250629 01:51:57.466573 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 556c2050d2cb40f28a3a0bfb9d677d69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 556c2050d2cb40f28a3a0bfb9d677d69 already in progress: creating tablet
I20250629 01:51:57.466758 710 leader_election.cc:304] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.467130 2337 raft_consensus.cc:2802] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:57.467008 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1097) took 1685 ms. Trace:
I20250629 01:51:57.467269 2337 raft_consensus.cc:695] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:57.467728 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9cfff25bea764e62954e49611b71703c (table default.Table_bf7b726d4c38474994e10c22d1aa6076 [id=875222c495154458ae2052f4ca5d220f]) successfully deleted
I20250629 01:51:57.467730 2337 consensus_queue.cc:237] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.468117 710 leader_election.cc:304] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.468434 2049 raft_consensus.cc:2802] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:57.468597 2049 raft_consensus.cc:695] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:57.468780 2049 consensus_queue.cc:237] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.471509 1158 ts_tablet_manager.cc:1918] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.471596 1158 log.cc:1199] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/258994dd24804f4aaa5e299436fac9b0
I20250629 01:51:57.471832 1158 ts_tablet_manager.cc:1939] T 258994dd24804f4aaa5e299436fac9b0 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:57.481034 1156 rpcz_store.cc:276] 0629 01:51:55.781874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.781921 (+ 47us) service_pool.cc:225] Handling call
0629 01:51:57.466627 (+1684706us) ts_tablet_manager.cc:1145] Deleting tablet d54bdcf7e1a14c669472af8acf36aac2
0629 01:51:57.466636 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.466949 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1684522}
I20250629 01:51:57.481987 1156 tablet_replica.cc:331] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.482169 1156 raft_consensus.cc:2241] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
W20250629 01:51:57.482267 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d54bdcf7e1a14c669472af8acf36aac2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d54bdcf7e1a14c669472af8acf36aac2 already in progress: creating tablet
I20250629 01:51:57.482484 1156 raft_consensus.cc:2270] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.483037 1156 ts_tablet_manager.cc:1905] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.489465 2404 tablet.cc:1722] T ddc24d0e668c49ac825b2578849ab2a3 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:57.508339 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af4707cebd704b4ba5a48ab600ab26f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.508844 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af4707cebd704b4ba5a48ab600ab26f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:57.509977 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1ca655a041b4aff95df14041a51b6cf consensus state has no leader
I20250629 01:51:57.507995 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_40a12cf7daec42ef90b22c45964e56f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.511736 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40a12cf7daec42ef90b22c45964e56f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.513741 2336 raft_consensus.cc:491] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.513877 2336 raft_consensus.cc:513] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:57.513979 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77d497c0dc4f49caa22758d3d7517a95 consensus state has no leader
W20250629 01:51:57.514361 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1b5deb7f2204e3aa271ddd818a360d3 consensus state has no leader
I20250629 01:51:57.514492 2336 leader_election.cc:290] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.514890 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 49844d26db6447de8c323376c7756437
I20250629 01:51:57.515306 446 catalog_manager.cc:6652] ReplaceTablet: tablet 49844d26db6447de8c323376c7756437 deleted and replaced by tablet f682b52dce6a4a318db9ef05431197b4
I20250629 01:51:57.516566 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 49844d26db6447de8c323376c7756437 with delete_type TABLET_DATA_DELETED (replaced by tablet f682b52dce6a4a318db9ef05431197b4) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.516973 2336 raft_consensus.cc:491] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.517103 2336 raft_consensus.cc:513] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.517647 2336 leader_election.cc:290] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.518325 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648b0a0f742b4e7da9eaff6bd8795fc4" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.518791 712 leader_election.cc:343] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.518325 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.519227 712 leader_election.cc:343] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.523844 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:57.524036 1052 raft_consensus.cc:2466] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:57.528342 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d387af53209147a6bac7afeda9a411ab
I20250629 01:51:57.528965 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 67e90a48af8f48bba3b3fd4f4110d367 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:57.529234 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d387af53209147a6bac7afeda9a411ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.529649 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9c90ec6864664fe48d8b9a92837b748c
I20250629 01:51:57.530530 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c90ec6864664fe48d8b9a92837b748c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.531165 884 tablet_service.cc:1515] Processing DeleteTablet for tablet d387af53209147a6bac7afeda9a411ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.531358 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c90ec6864664fe48d8b9a92837b748c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.531978 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet d387af53209147a6bac7afeda9a411ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d387af53209147a6bac7afeda9a411ab already in progress: creating tablet
W20250629 01:51:57.532253 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9c90ec6864664fe48d8b9a92837b748c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c90ec6864664fe48d8b9a92837b748c already in progress: creating tablet
I20250629 01:51:57.533833 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 82a837932f14419c8b32fcd2c9358215 (table default.Table_3592b5e7e0c44766a622a04a3a5c9b03 [id=ed3c8fa707924f8ea53d590a7f9b3d68]). Expected version 2 got 0
I20250629 01:51:57.533998 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 95247cc778504f4d85d9013f84c59e69 (table default.Table_355a96774d4f4e0e9f5835f7a7450e0c [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]). Expected version 1 got 0
I20250629 01:51:57.534475 1156 ts_tablet_manager.cc:1918] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:57.535125 1156 log.cc:1199] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/add482b533d045e7a3010064fae730cf
I20250629 01:51:57.535732 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_aa8289c495a049b59d261595709a2994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:57.535833 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fafd19402c4f4949ba4b7a4bf44473aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.536209 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa8289c495a049b59d261595709a2994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.536413 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_90112469ff66420a85ce68ec4cc69b56" } modify_external_catalogs: true
I20250629 01:51:57.536547 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_90112469ff66420a85ce68ec4cc69b56" } modify_external_catalogs: true
W20250629 01:51:57.537582 712 consensus_peers.cc:489] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:57.537909 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.536209 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fafd19402c4f4949ba4b7a4bf44473aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.538118 1157 tablet_replica.cc:331] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:57.538342 1157 raft_consensus.cc:2241] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.538537 1157 raft_consensus.cc:2270] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.539271 1157 ts_tablet_manager.cc:1905] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.540071 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_86157cacc72642338c2bff2ec0dd2052" } new_table_name: "default.Table_20b2067750ab47deae63c0f954ef7683" modify_external_catalogs: true
I20250629 01:51:57.540668 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d5002228f791426d81f36cbba806a047" } modify_external_catalogs: true
I20250629 01:51:57.545276 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 258994dd24804f4aaa5e299436fac9b0 (table default.Table_a8bede9e061941eaabacba34109b16ab [id=c4001552f202478e92afae5c3aacfa9b]) successfully deleted
I20250629 01:51:57.545859 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb with delete_type TABLET_DATA_DELETED (replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.545972 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.547106 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.547186 1157 ts_tablet_manager.cc:1918] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.547394 1157 log.cc:1199] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/6f3a65c9c1c740b1865b96a6e2aea524
I20250629 01:51:57.547959 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d5002228f791426d81f36cbba806a047" } modify_external_catalogs: true
I20250629 01:51:57.548127 1157 ts_tablet_manager.cc:1939] T 6f3a65c9c1c740b1865b96a6e2aea524 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:57.548452 2412 raft_consensus.cc:491] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.548564 2412 raft_consensus.cc:513] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.548774 449 catalog_manager.cc:5582] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:57.549099 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 82a837932f14419c8b32fcd2c9358215 (table default.Table_3592b5e7e0c44766a622a04a3a5c9b03 [id=ed3c8fa707924f8ea53d590a7f9b3d68]). Expected version 2 got 0
I20250629 01:51:57.549127 2412 leader_election.cc:290] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.549350 449 catalog_manager.cc:5582] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:57.549556 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cb7314fbdfc74b1c8d9d1be9c05244cb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.549738 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
W20250629 01:51:57.549923 712 leader_election.cc:343] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.550987 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 6f3a65c9c1c740b1865b96a6e2aea524 (table default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]) successfully deleted
I20250629 01:51:57.552021 1156 ts_tablet_manager.cc:1939] T add482b533d045e7a3010064fae730cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:57.553967 710 leader_election.cc:343] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0d4320f615464fedbcab25137952f684
I20250629 01:51:57.554051 710 leader_election.cc:304] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:57.554250 2433 raft_consensus.cc:2747] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.556604 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1094) took 1775 ms. Trace:
I20250629 01:51:57.557179 1156 rpcz_store.cc:276] 0629 01:51:55.781024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.782106 (+ 1082us) service_pool.cc:225] Handling call
0629 01:51:57.481948 (+1699842us) ts_tablet_manager.cc:1145] Deleting tablet add482b533d045e7a3010064fae730cf
0629 01:51:57.481957 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.509913 (+ 27956us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.533995 (+ 24082us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.556483 (+ 22488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1699690}
I20250629 01:51:57.558750 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet add482b533d045e7a3010064fae730cf (table default.Table_c08e69cfa0894f67beafee56aa098ad9 [id=1f28cdb4b4d34d6cb5880828f17011ed]) successfully deleted
I20250629 01:51:57.559656 1156 tablet_replica.cc:331] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.560719 1156 raft_consensus.cc:2241] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:57.561862 710 leader_election.cc:304] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:51:57.562366 710 leader_election.cc:343] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 648b0a0f742b4e7da9eaff6bd8795fc4
I20250629 01:51:57.562480 710 leader_election.cc:304] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:57.562908 2412 raft_consensus.cc:2747] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.562588 2049 raft_consensus.cc:2802] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:57.563446 2049 raft_consensus.cc:491] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:57.563609 2049 raft_consensus.cc:3058] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.564116 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 49844d26db6447de8c323376c7756437 with delete_type TABLET_DATA_DELETED (replaced by tablet f682b52dce6a4a318db9ef05431197b4) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.564271 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 49844d26db6447de8c323376c7756437 with delete_type TABLET_DATA_DELETED (replaced by tablet f682b52dce6a4a318db9ef05431197b4) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.564312 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d387af53209147a6bac7afeda9a411ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.562373 1156 pending_rounds.cc:70] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:57.565315 1156 pending_rounds.cc:77] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759203732705280 op_type: NO_OP noop_request { }
I20250629 01:51:57.565972 1156 raft_consensus.cc:2887] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:57.566537 1156 raft_consensus.cc:2270] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:57.570248 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 49844d26db6447de8c323376c7756437 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49844d26db6447de8c323376c7756437 already in progress: creating tablet
I20250629 01:51:57.570887 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c90ec6864664fe48d8b9a92837b748c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.571535 1156 ts_tablet_manager.cc:1905] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.590498 2049 raft_consensus.cc:513] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.595906 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:57.596441 712 leader_election.cc:343] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.597136 2049 leader_election.cc:290] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.598837 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_57567ad8836042d783d3e6be3438923e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.599362 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57567ad8836042d783d3e6be3438923e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.599565 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_681f0348d61c4df39b9e7a9806daeaca" } new_table_name: "default.Table_89c922281dde4b02ae3a2875c9ff73ef" modify_external_catalogs: true
I20250629 01:51:57.598891 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4bc29d1ce5c842abb7b4c888bedc32c3" } modify_external_catalogs: true
I20250629 01:51:57.602553 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4bc29d1ce5c842abb7b4c888bedc32c3" } modify_external_catalogs: true
I20250629 01:51:57.603039 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_aa685973eace4bf5ab1a4d4cbaef51c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.603399 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa685973eace4bf5ab1a4d4cbaef51c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.604085 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 4de6fca85fb643b998be42e9bc7bdbb6 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:57.605180 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a40c1a8eec9549a8bdf695d6516440a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.605420 1157 tablet_replica.cc:331] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:57.605631 1157 raft_consensus.cc:2241] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.605767 1157 raft_consensus.cc:2270] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.605921 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3a65c9c1c740b1865b96a6e2aea524 with delete_type TABLET_DATA_DELETED (replaced by tablet 5ab0aa4e74d345b2a214b90b5018c82b) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.606457 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 49844d26db6447de8c323376c7756437 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49844d26db6447de8c323376c7756437 already in progress: creating tablet
I20250629 01:51:57.606550 1154 tablet_bootstrap.cc:492] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.606606 1157 ts_tablet_manager.cc:1905] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.606649 1154 ts_tablet_manager.cc:1397] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.161s user 0.019s sys 0.023s
W20250629 01:51:57.606825 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet d387af53209147a6bac7afeda9a411ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d387af53209147a6bac7afeda9a411ab already in progress: creating tablet
W20250629 01:51:57.607144 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9c90ec6864664fe48d8b9a92837b748c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c90ec6864664fe48d8b9a92837b748c already in progress: creating tablet
I20250629 01:51:57.607321 1154 raft_consensus.cc:357] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.607476 1154 raft_consensus.cc:383] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.607543 1154 raft_consensus.cc:738] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.607764 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:57.607733 1154 consensus_queue.cc:260] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.607880 1048 raft_consensus.cc:3058] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.608345 1154 ts_tablet_manager.cc:1428] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:57.608592 1154 tablet_bootstrap.cc:492] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.609851 1154 tablet_bootstrap.cc:654] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.610435 1048 raft_consensus.cc:2466] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:51:57.612536 710 leader_election.cc:343] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:51:57.612623 710 leader_election.cc:304] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:57.612852 2332 raft_consensus.cc:2747] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.613343 2371 tablet.cc:1722] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:57.615047 1157 ts_tablet_manager.cc:1918] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.615168 1157 log.cc:1199] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a40c1a8eec9549a8bdf695d6516440a8
I20250629 01:51:57.615409 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 027d08ece22d4b97a3c31e68bcaa396a
I20250629 01:51:57.615545 1157 ts_tablet_manager.cc:1939] T a40c1a8eec9549a8bdf695d6516440a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:57.616068 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4622cfe68f7a49a8a6d84d12a686dea7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.616596 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4622cfe68f7a49a8a6d84d12a686dea7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.617728 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a40c1a8eec9549a8bdf695d6516440a8 (table default.Table_6f6d8cf186fc4934ac5690bac0754373 [id=db3232e7e81240a699e87310dc0627d3]) successfully deleted
W20250629 01:51:57.617933 442 catalog_manager.cc:4895] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6f3a65c9c1c740b1865b96a6e2aea524 because the tablet was not found. No further retry: Not found: Tablet not found: 6f3a65c9c1c740b1865b96a6e2aea524
I20250629 01:51:57.619033 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 027d08ece22d4b97a3c31e68bcaa396a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.620980 447 master_service.cc:948] ReplaceTablet: received request to replace tablet b6d60a3c1bc642969ced8c712e5af1c1 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:57.623684 1154 tablet_bootstrap.cc:492] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.623857 1154 ts_tablet_manager.cc:1397] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20250629 01:51:57.624843 1154 raft_consensus.cc:357] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.625052 1154 raft_consensus.cc:383] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.625140 1154 raft_consensus.cc:738] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.625375 1154 consensus_queue.cc:260] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.625999 1154 ts_tablet_manager.cc:1428] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:57.626307 1154 tablet_bootstrap.cc:492] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.627983 1154 tablet_bootstrap.cc:654] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.628651 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 027d08ece22d4b97a3c31e68bcaa396a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.629272 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 027d08ece22d4b97a3c31e68bcaa396a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 027d08ece22d4b97a3c31e68bcaa396a already in progress: creating tablet
I20250629 01:51:57.633491 1156 ts_tablet_manager.cc:1918] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:57.634093 1156 log.cc:1199] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/d6bf3c7d0bb548e0b77ba8b569a72d0d
I20250629 01:51:57.634987 710 leader_election.cc:304] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:57.635223 2433 raft_consensus.cc:2802] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:57.635612 1154 tablet_bootstrap.cc:492] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.635643 2433 raft_consensus.cc:695] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:57.635754 1154 ts_tablet_manager.cc:1397] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20250629 01:51:57.636184 2433 consensus_queue.cc:237] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.636377 1154 raft_consensus.cc:357] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.636577 1154 raft_consensus.cc:383] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.636651 1154 raft_consensus.cc:738] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.636840 1154 consensus_queue.cc:260] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.636958 2383 tablet.cc:1722] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:57.637370 1154 ts_tablet_manager.cc:1428] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:57.637812 1154 tablet_bootstrap.cc:492] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.638940 2049 consensus_queue.cc:1035] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:57.639191 1154 tablet_bootstrap.cc:654] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.641217 2049 raft_consensus.cc:491] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.641520 2049 raft_consensus.cc:513] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.642865 2049 leader_election.cc:290] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.643095 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8a880c9bb4b94d3b9b12f28d48565f06" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.644214 712 leader_election.cc:343] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:57.647511 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bf612fea5b74a0bb78700cc07c4d68d consensus state has no leader
I20250629 01:51:57.647825 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e3e8a33e42a84ab8a6909aa7048291f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:57.649078 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2660c211d65f41b0a50b5a6f7d8be48a
I20250629 01:51:57.649272 1156 ts_tablet_manager.cc:1939] T d6bf3c7d0bb548e0b77ba8b569a72d0d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.649636 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 28da32ab17024f519105c2490efd9136 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:57.650496 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1093) took 1872 ms. Trace:
W20250629 01:51:57.650842 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e8a33e42a84ab8a6909aa7048291f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.651340 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 027d08ece22d4b97a3c31e68bcaa396a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.651002 1156 rpcz_store.cc:276] 0629 01:51:55.777575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:55.781300 (+ 3725us) service_pool.cc:225] Handling call
0629 01:51:57.559384 (+1778084us) ts_tablet_manager.cc:1145] Deleting tablet d6bf3c7d0bb548e0b77ba8b569a72d0d
0629 01:51:57.559398 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.619031 (+ 59633us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.632975 (+ 13944us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.650376 (+ 17401us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23,"tablet-delete.queue_time_us":1776850}
I20250629 01:51:57.652467 1156 tablet_replica.cc:331] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.652817 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]) successfully deleted
I20250629 01:51:57.653213 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 2660c211d65f41b0a50b5a6f7d8be48a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.653263 1156 raft_consensus.cc:2241] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.653752 1156 raft_consensus.cc:2270] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.655633 1156 ts_tablet_manager.cc:1905] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.656385 1155 tablet_bootstrap.cc:492] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.656540 1155 ts_tablet_manager.cc:1397] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.214s user 0.002s sys 0.079s
I20250629 01:51:57.657277 1155 raft_consensus.cc:357] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.657464 1155 raft_consensus.cc:383] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.657565 1155 raft_consensus.cc:738] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.657763 1155 consensus_queue.cc:260] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.658241 1155 ts_tablet_manager.cc:1428] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:57.658525 1155 tablet_bootstrap.cc:492] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:57.659143 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 548674f1507743228fcd04bfd91500ae consensus state has no leader
I20250629 01:51:57.659988 1155 tablet_bootstrap.cc:654] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.660185 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 2660c211d65f41b0a50b5a6f7d8be48a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.662791 1154 tablet_bootstrap.cc:492] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.663059 1154 ts_tablet_manager.cc:1397] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250629 01:51:57.664330 1154 raft_consensus.cc:357] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.664603 1154 raft_consensus.cc:383] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.664708 1154 raft_consensus.cc:738] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.665074 1154 consensus_queue.cc:260] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.665940 1154 ts_tablet_manager.cc:1428] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:57.666375 1154 tablet_bootstrap.cc:492] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.666393 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8d85b9a0e5854e9e9fc16b98143c1cc0
I20250629 01:51:57.666993 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4de6fca85fb643b998be42e9bc7bdbb6
I20250629 01:51:57.667388 455 catalog_manager.cc:6652] ReplaceTablet: tablet 4de6fca85fb643b998be42e9bc7bdbb6 deleted and replaced by tablet 332a2808e00c4c52afa3931cb8c53516
I20250629 01:51:57.668468 1155 tablet_bootstrap.cc:492] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.668637 1155 ts_tablet_manager.cc:1397] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250629 01:51:57.669276 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b6d60a3c1bc642969ced8c712e5af1c1
I20250629 01:51:57.669245 1155 raft_consensus.cc:357] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.669555 1155 raft_consensus.cc:383] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.669600 1154 tablet_bootstrap.cc:654] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.669654 1155 raft_consensus.cc:738] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.669720 447 catalog_manager.cc:6652] ReplaceTablet: tablet b6d60a3c1bc642969ced8c712e5af1c1 deleted and replaced by tablet 24e9caef62f64a4281789aba8f83386a
I20250629 01:51:57.669937 1155 consensus_queue.cc:260] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.670420 1155 ts_tablet_manager.cc:1428] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:57.670711 1155 tablet_bootstrap.cc:492] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.678550 1155 tablet_bootstrap.cc:654] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.688349 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.689424 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4de6fca85fb643b998be42e9bc7bdbb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 332a2808e00c4c52afa3931cb8c53516) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.690146 749 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d60a3c1bc642969ced8c712e5af1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 24e9caef62f64a4281789aba8f83386a) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.660734 1159 tablet_bootstrap.cc:492] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.691126 1159 ts_tablet_manager.cc:1397] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.257s user 0.001s sys 0.087s
I20250629 01:51:57.691745 1159 raft_consensus.cc:357] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.691877 1159 raft_consensus.cc:383] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.691922 1159 raft_consensus.cc:738] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.692106 1159 consensus_queue.cc:260] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.692515 1159 ts_tablet_manager.cc:1428] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:57.692724 1159 tablet_bootstrap.cc:492] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.694674 1159 tablet_bootstrap.cc:654] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.668252 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 4de6fca85fb643b998be42e9bc7bdbb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 332a2808e00c4c52afa3931cb8c53516) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.668371 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.670522 890 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d60a3c1bc642969ced8c712e5af1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 24e9caef62f64a4281789aba8f83386a) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.697302 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4de6fca85fb643b998be42e9bc7bdbb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4de6fca85fb643b998be42e9bc7bdbb6 already in progress: creating tablet
W20250629 01:51:57.697635 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 already in progress: creating tablet
W20250629 01:51:57.697844 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b6d60a3c1bc642969ced8c712e5af1c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d60a3c1bc642969ced8c712e5af1c1 already in progress: creating tablet
W20250629 01:51:57.709100 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2660c211d65f41b0a50b5a6f7d8be48a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2660c211d65f41b0a50b5a6f7d8be48a already in progress: creating tablet
I20250629 01:51:57.712889 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14435d8c8f2846e393084f1fa1f7f303" } new_table_name: "default.Table_5d44781f81e640699c86c7c38ad4705b" modify_external_catalogs: true
I20250629 01:51:57.716974 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4ae5ee4f0ebe46d7beb3958a50f0dd87" } new_table_name: "default.Table_2fcd67db9bb141f8a0447eb7e7ca8f63" modify_external_catalogs: true
I20250629 01:51:57.718576 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eefef9f944a5441b8fdc130d0ab579a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.719499 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eefef9f944a5441b8fdc130d0ab579a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:57.723230 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16342cedfef24b7e9082e9f1f9f32b5d consensus state has no leader
W20250629 01:51:57.723824 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 027d08ece22d4b97a3c31e68bcaa396a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 027d08ece22d4b97a3c31e68bcaa396a already in progress: creating tablet
I20250629 01:51:57.724161 2363 raft_consensus.cc:491] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.724273 2363 raft_consensus.cc:513] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.724777 2363 leader_election.cc:290] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:57.725422 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76118a6e684448b09f244d44c909ccaa consensus state has no leader
I20250629 01:51:57.725673 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06e31520d3ad4c3395a59049755a4725" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:57.726210 986 leader_election.cc:343] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.726470 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06e31520d3ad4c3395a59049755a4725" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.726595 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_355a96774d4f4e0e9f5835f7a7450e0c" } new_table_name: "default.Table_20ba093e340e4fe7ae570398e255a99b" modify_external_catalogs: true
W20250629 01:51:57.726873 988 leader_election.cc:343] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.726963 988 leader_election.cc:304] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.727293 2363 raft_consensus.cc:2747] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.727946 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2660c211d65f41b0a50b5a6f7d8be48a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.727984 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.728353 1158 tablet_replica.cc:331] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.728515 1158 raft_consensus.cc:2241] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.728636 1158 raft_consensus.cc:2270] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.729072 1158 ts_tablet_manager.cc:1905] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.730388 1159 tablet_bootstrap.cc:492] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.730485 1159 ts_tablet_manager.cc:1397] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20250629 01:51:57.731107 1159 raft_consensus.cc:357] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.731235 1159 raft_consensus.cc:383] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.731271 1159 raft_consensus.cc:738] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.731436 1159 consensus_queue.cc:260] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.734699 1159 ts_tablet_manager.cc:1428] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:51:57.734922 1159 tablet_bootstrap.cc:492] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.740240 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1f13038d42254b24858c6b4d8af9d70a" } modify_external_catalogs: true
I20250629 01:51:57.740375 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1f13038d42254b24858c6b4d8af9d70a" } modify_external_catalogs: true
I20250629 01:51:57.744266 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cd7226b7561e4b56a60721679781b7a6" } modify_external_catalogs: true
I20250629 01:51:57.744421 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cd7226b7561e4b56a60721679781b7a6" } modify_external_catalogs: true
I20250629 01:51:57.747511 1158 ts_tablet_manager.cc:1918] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.747603 1158 log.cc:1199] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/74b3badd684d45fa88dc8cbcfc61001a
I20250629 01:51:57.747865 1158 ts_tablet_manager.cc:1939] T 74b3badd684d45fa88dc8cbcfc61001a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:57.749146 1156 ts_tablet_manager.cc:1918] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.749356 1156 log.cc:1199] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0ba9eabd06174b87a1485f4921895c5f
I20250629 01:51:57.749818 1156 ts_tablet_manager.cc:1939] T 0ba9eabd06174b87a1485f4921895c5f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.750923 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1140) took 1694 ms. Trace:
I20250629 01:51:57.751060 1156 rpcz_store.cc:276] 0629 01:51:56.056452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.103581 (+ 47129us) service_pool.cc:225] Handling call
0629 01:51:57.652315 (+1548734us) ts_tablet_manager.cc:1145] Deleting tablet 0ba9eabd06174b87a1485f4921895c5f
0629 01:51:57.652327 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.689008 (+ 36681us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.748974 (+ 59966us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.750831 (+ 1857us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1548501}
I20250629 01:51:57.751749 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0ba9eabd06174b87a1485f4921895c5f (table default.Table_6e4d952c729c4984a21f16da686e28f0 [id=2a01904ec4bd4b48b528d7c0cfca4a43]) successfully deleted
I20250629 01:51:57.755160 1156 tablet_replica.cc:331] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:57.755218 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95247cc778504f4d85d9013f84c59e69 consensus state has no leader
I20250629 01:51:57.755414 1156 raft_consensus.cc:2241] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.758298 1159 tablet_bootstrap.cc:654] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.760344 1156 raft_consensus.cc:2270] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.762902 447 catalog_manager.cc:5582] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:57.763541 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:57.764318 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 28da32ab17024f519105c2490efd9136
I20250629 01:51:57.764671 454 catalog_manager.cc:6652] ReplaceTablet: tablet 28da32ab17024f519105c2490efd9136 deleted and replaced by tablet 198d774683a34021be94b8057a5ae5ff
I20250629 01:51:57.763933 447 catalog_manager.cc:5582] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:57.766099 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:57.766789 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 58e5cb8c50734d9fbb2b09e764deb11b
I20250629 01:51:57.767268 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 58e5cb8c50734d9fbb2b09e764deb11b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.767788 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 58e5cb8c50734d9fbb2b09e764deb11b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58e5cb8c50734d9fbb2b09e764deb11b already in progress: creating tablet
I20250629 01:51:57.768489 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 28da32ab17024f519105c2490efd9136 with delete_type TABLET_DATA_DELETED (replaced by tablet 198d774683a34021be94b8057a5ae5ff) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.766413 447 catalog_manager.cc:5582] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:57.768700 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 58e5cb8c50734d9fbb2b09e764deb11b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.768942 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943]). Expected version 5 got 0
I20250629 01:51:57.769754 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 28da32ab17024f519105c2490efd9136 with delete_type TABLET_DATA_DELETED (replaced by tablet 198d774683a34021be94b8057a5ae5ff) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:57.770258 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 28da32ab17024f519105c2490efd9136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28da32ab17024f519105c2490efd9136 already in progress: creating tablet
I20250629 01:51:57.775583 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_4d28e4ec92114a31856c3992b0728729 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 5 got 0
I20250629 01:51:57.785960 447 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_4d28e4ec92114a31856c3992b0728729 [id=3d577c959e5b4549a57fa4cdc320a0d6] alter complete (version 5)
I20250629 01:51:57.786931 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cfff25bea764e62954e49611b71703c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.787354 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e31520d3ad4c3395a59049755a4725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.787570 749 tablet_service.cc:1515] Processing DeleteTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.790227 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 58e5cb8c50734d9fbb2b09e764deb11b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.790254 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]). Expected version 1 got 0
I20250629 01:51:57.790974 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.791141 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4de6fca85fb643b998be42e9bc7bdbb6 with delete_type TABLET_DATA_DELETED (replaced by tablet 332a2808e00c4c52afa3931cb8c53516) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.791262 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet b6d60a3c1bc642969ced8c712e5af1c1 with delete_type TABLET_DATA_DELETED (replaced by tablet 24e9caef62f64a4281789aba8f83386a) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.791366 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 28da32ab17024f519105c2490efd9136 with delete_type TABLET_DATA_DELETED (replaced by tablet 198d774683a34021be94b8057a5ae5ff) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.791689 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b94f8250a143b096703f2b0386e3cd with delete_type TABLET_DATA_DELETED (replaced by tablet c5771fdd618e45118df2b6dab7f668d9) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:57.794579 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:51:57.794914 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:51:57.795192 453 catalog_manager.cc:5582] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:57.795527 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 95247cc778504f4d85d9013f84c59e69 (table default.Table_20ba093e340e4fe7ae570398e255a99b [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]). Expected version 2 got 0
I20250629 01:51:57.796242 2363 raft_consensus.cc:491] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.796370 2363 raft_consensus.cc:513] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.796909 2363 leader_election.cc:290] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.797362 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d9ef77181ba48f3a6de16b945c8df1a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.797752 988 leader_election.cc:343] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.797747 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d9ef77181ba48f3a6de16b945c8df1a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:57.798183 986 leader_election.cc:343] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.798269 986 leader_election.cc:304] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.798605 2363 raft_consensus.cc:2747] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.800163 1159 tablet_bootstrap.cc:492] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.800405 1159 ts_tablet_manager.cc:1397] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20250629 01:51:57.800798 1156 ts_tablet_manager.cc:1905] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.801251 2363 raft_consensus.cc:491] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.801353 2363 raft_consensus.cc:513] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.801755 2363 leader_election.cc:290] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.802592 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_246987d15c034ab5b4be46b8ec5426be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:57.802773 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f216b85d57b04573a5ec1eb86e843273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.803048 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_246987d15c034ab5b4be46b8ec5426be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:57.803102 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f216b85d57b04573a5ec1eb86e843273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.803552 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6092e19acf974f2dad5a903da72944ba" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.804052 988 leader_election.cc:343] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.804262 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c0c3a2b64e794d63a3c9bce071cf5b97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.804742 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0c3a2b64e794d63a3c9bce071cf5b97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:57.806689 712 consensus_peers.cc:489] T 38f831a6dd68493382d5f2920d0794a8 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:57.806864 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_44f5de58c2f14c7b8c1981d2f735bccd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.807291 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44f5de58c2f14c7b8c1981d2f735bccd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.809530 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2adf0270dfed47439f6cf4e4c956e185 (DEFAULT_TABLE table=default.Table_dd06d636d7c6466599c047ba9ccbde72 [id=69e8ffd30840420998053fa803658d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.809839 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adf0270dfed47439f6cf4e4c956e185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.820981 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_72800b006f6e4b2eab6c57ecea71aef0" } modify_external_catalogs: true
I20250629 01:51:57.821159 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_72800b006f6e4b2eab6c57ecea71aef0" } modify_external_catalogs: true
I20250629 01:51:57.823679 1158 tablet_replica.cc:331] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:57.824115 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1382661b867943bc8431964543552b18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.824512 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1382661b867943bc8431964543552b18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.823890 1158 raft_consensus.cc:2241] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.825853 1158 raft_consensus.cc:2270] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:57.826946 1158 ts_tablet_manager.cc:1905] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:57.830675 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 6092e19acf974f2dad5a903da72944ba from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:57.831014 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6092e19acf974f2dad5a903da72944ba" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:57.831488 986 leader_election.cc:343] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.831588 986 leader_election.cc:304] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.831881 2363 raft_consensus.cc:2747] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:57.834260 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2660c211d65f41b0a50b5a6f7d8be48a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2660c211d65f41b0a50b5a6f7d8be48a already in progress: creating tablet
I20250629 01:51:57.834434 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 74b3badd684d45fa88dc8cbcfc61001a (table default.Table_75f56cc1109c41f2a0a3ba83902b88ea [id=fcdd225098994f2f8c425a8fc3165d24]) successfully deleted
I20250629 01:51:57.835592 1159 raft_consensus.cc:357] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.835716 1159 raft_consensus.cc:383] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.835754 1159 raft_consensus.cc:738] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.835889 1159 consensus_queue.cc:260] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.837836 963 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:57.841063 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 2adf0270dfed47439f6cf4e4c956e185 (DEFAULT_TABLE table=default.Table_dd06d636d7c6466599c047ba9ccbde72 [id=69e8ffd30840420998053fa803658d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.841372 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adf0270dfed47439f6cf4e4c956e185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.843779 775 raft_consensus.cc:1273] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250629 01:51:57.844368 988 consensus_peers.cc:489] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:57.844578 2374 consensus_queue.cc:1035] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:57.845513 1154 tablet_bootstrap.cc:492] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.845667 1154 ts_tablet_manager.cc:1397] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.179s user 0.047s sys 0.032s
I20250629 01:51:57.846047 890 tablet_service.cc:1468] Processing CreateTablet for tablet 50f2d76bc0a041c79b058c537150b894 (DEFAULT_TABLE table=default.Table_d96f54aeed554f3bbe79066de62debc5 [id=7a82929ed99342f599aff24655e5e704]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:57.846643 710 leader_election.cc:343] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:51:57.846774 710 leader_election.cc:304] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:57.846897 1154 raft_consensus.cc:357] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.847039 1154 raft_consensus.cc:383] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.847084 1154 raft_consensus.cc:738] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.847069 2436 raft_consensus.cc:2747] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.847254 1154 consensus_queue.cc:260] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:57.847586 712 consensus_peers.cc:489] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:57.847694 1154 ts_tablet_manager.cc:1428] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:57.847908 1154 tablet_bootstrap.cc:492] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:57.848531 712 consensus_peers.cc:489] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:57.851485 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0a878b9a2af948319e9adaf6783959a0" } modify_external_catalogs: true
I20250629 01:51:57.851663 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0a878b9a2af948319e9adaf6783959a0" } modify_external_catalogs: true
I20250629 01:51:57.852880 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2adf0270dfed47439f6cf4e4c956e185 (DEFAULT_TABLE table=default.Table_dd06d636d7c6466599c047ba9ccbde72 [id=69e8ffd30840420998053fa803658d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.853282 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2adf0270dfed47439f6cf4e4c956e185. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.853478 753 tablet_service.cc:1468] Processing CreateTablet for tablet cdebad5cfed047a2ad4f58bdef3c105f (DEFAULT_TABLE table=default.Table_bac468da747c48e4bf6c7884bbd9e1f1 [id=5fe17a5d54304ae5ae72c6cde593538e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.853965 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdebad5cfed047a2ad4f58bdef3c105f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.857143 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fc9218fdbd474925bd2444d014e3ba06" } new_table_name: "default.Table_4a3390362d0b4fa3ac770487937283b7" modify_external_catalogs: true
I20250629 01:51:57.863855 877 tablet_service.cc:1468] Processing CreateTablet for tablet cdebad5cfed047a2ad4f58bdef3c105f (DEFAULT_TABLE table=default.Table_bac468da747c48e4bf6c7884bbd9e1f1 [id=5fe17a5d54304ae5ae72c6cde593538e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.864235 2418 raft_consensus.cc:491] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.864843 2418 raft_consensus.cc:513] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.865353 2418 leader_election.cc:290] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:57.865900 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:57.867722 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "176bd3e921c74f9a93d4b2f1c0fc97a7" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:57.868146 850 leader_election.cc:304] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.868335 1154 tablet_bootstrap.cc:654] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.868496 2418 raft_consensus.cc:2747] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.871450 1154 tablet_bootstrap.cc:492] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.871668 1154 ts_tablet_manager.cc:1397] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250629 01:51:57.872542 1154 raft_consensus.cc:357] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.872776 1154 raft_consensus.cc:383] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.872898 1154 raft_consensus.cc:738] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.873097 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f2336702c5e941ce994df6edb1970192
I20250629 01:51:57.873238 1154 consensus_queue.cc:260] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:57.873890 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f2336702c5e941ce994df6edb1970192 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.874044 1154 ts_tablet_manager.cc:1428] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:57.874323 1154 tablet_bootstrap.cc:492] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.875999 1154 tablet_bootstrap.cc:654] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.876607 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 45461dc364d847d0b0272ac714559008
I20250629 01:51:57.877410 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 45461dc364d847d0b0272ac714559008 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:57.877897 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f3b9149e73b24dac8af5e1d055f3f63d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:57.878512 1156 ts_tablet_manager.cc:1918] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.878607 1156 log.cc:1199] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/56b60431b72e4792aa1c32fdee40cdb9
I20250629 01:51:57.878882 1156 ts_tablet_manager.cc:1939] T 56b60431b72e4792aa1c32fdee40cdb9 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:57.882246 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b9149e73b24dac8af5e1d055f3f63d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.883395 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71c6625c776b4ab6a54a4c6d4d928cf2" } new_table_name: "default.Table_9d65bac066b8474f8f1166914053b2a4" modify_external_catalogs: true
I20250629 01:51:57.884339 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9823577beb5242a798718da814e438bf" } modify_external_catalogs: true
I20250629 01:51:57.884462 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1139) took 1828 ms. Trace:
I20250629 01:51:57.884483 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9823577beb5242a798718da814e438bf" } modify_external_catalogs: true
I20250629 01:51:57.884569 1156 rpcz_store.cc:276] 0629 01:51:56.056370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.103576 (+ 47206us) service_pool.cc:225] Handling call
0629 01:51:57.751518 (+1647942us) ts_tablet_manager.cc:1145] Deleting tablet 56b60431b72e4792aa1c32fdee40cdb9
0629 01:51:57.751533 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.833781 (+ 82248us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.878393 (+ 44612us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.884430 (+ 6037us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643292}
I20250629 01:51:57.884792 1154 tablet_bootstrap.cc:492] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:57.884919 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1141) took 1742 ms. Trace:
I20250629 01:51:57.884967 1154 ts_tablet_manager.cc:1397] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250629 01:51:57.885010 1156 rpcz_store.cc:276] 0629 01:51:56.142024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.142747 (+ 723us) service_pool.cc:225] Handling call
0629 01:51:57.884800 (+1742053us) ts_tablet_manager.cc:1145] Deleting tablet ce90c045115844b0b96143f91f1be1d4
0629 01:51:57.884810 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.884891 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741869}
I20250629 01:51:57.885321 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1142) took 1742 ms. Trace:
I20250629 01:51:57.885421 1156 rpcz_store.cc:276] 0629 01:51:56.142670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.142957 (+ 287us) service_pool.cc:225] Handling call
0629 01:51:57.885216 (+1742259us) ts_tablet_manager.cc:1145] Deleting tablet 8873b53d7e42492da64f7f677e05c6f1
0629 01:51:57.885225 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.885297 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741876}
I20250629 01:51:57.885699 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1143) took 1735 ms. Trace:
I20250629 01:51:57.885793 1156 rpcz_store.cc:276] 0629 01:51:56.150682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.150762 (+ 80us) service_pool.cc:225] Handling call
0629 01:51:57.885594 (+1734832us) ts_tablet_manager.cc:1145] Deleting tablet 2e9e6727c5e3422ab1463298cf3ec974
0629 01:51:57.885603 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.885665 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1734698}
I20250629 01:51:57.886070 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1144) took 1681 ms. Trace:
I20250629 01:51:57.885954 1154 raft_consensus.cc:357] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.886149 1154 raft_consensus.cc:383] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.886159 1156 rpcz_store.cc:276] 0629 01:51:56.204362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.273455 (+ 69093us) service_pool.cc:225] Handling call
0629 01:51:57.885972 (+1612517us) ts_tablet_manager.cc:1145] Deleting tablet 9f6fbefddf434f32ab373b4e0191cee2
0629 01:51:57.885979 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.886041 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1612342}
I20250629 01:51:57.886220 1154 raft_consensus.cc:738] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.886775 1156 tablet_replica.cc:331] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:57.886986 744 tablet_service.cc:1515] Processing DeleteTablet for tablet f2336702c5e941ce994df6edb1970192 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.887424 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 45461dc364d847d0b0272ac714559008 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.887756 1156 raft_consensus.cc:2241] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:57.887843 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 56b60431b72e4792aa1c32fdee40cdb9 (table default.Table_07e9a307bb3344089bc369d3f1bf6c34 [id=7749e44179bf4424bbcb595803f61dae]) successfully deleted
I20250629 01:51:57.887866 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8844355e9b1144cd9cb3b2787a80e01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:57.888319 1156 raft_consensus.cc:2270] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:57.888669 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8844355e9b1144cd9cb3b2787a80e01c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:57.888216 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ce90c045115844b0b96143f91f1be1d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce90c045115844b0b96143f91f1be1d4 already in progress: creating tablet
W20250629 01:51:57.889418 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8873b53d7e42492da64f7f677e05c6f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8873b53d7e42492da64f7f677e05c6f1 already in progress: creating tablet
I20250629 01:51:57.889842 1156 ts_tablet_manager.cc:1905] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:57.890314 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2e9e6727c5e3422ab1463298cf3ec974 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e9e6727c5e3422ab1463298cf3ec974 already in progress: creating tablet
W20250629 01:51:57.890517 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9f6fbefddf434f32ab373b4e0191cee2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f6fbefddf434f32ab373b4e0191cee2 already in progress: creating tablet
I20250629 01:51:57.890614 2383 tablet.cc:1722] T 82a837932f14419c8b32fcd2c9358215 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:57.890901 1159 ts_tablet_manager.cc:1428] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.090s user 0.001s sys 0.000s
I20250629 01:51:57.894965 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d6fad403f1ae4427895f31b28a95daf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:57.895359 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6fad403f1ae4427895f31b28a95daf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:57.897426 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b] alter complete (version 1)
I20250629 01:51:57.898571 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.898928 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:57.899268 2371 tablet.cc:1722] T 82a837932f14419c8b32fcd2c9358215 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:57.908190 1154 consensus_queue.cc:260] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.908675 1154 ts_tablet_manager.cc:1428] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250629 01:51:57.908937 1154 tablet_bootstrap.cc:492] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:57.910281 1154 tablet_bootstrap.cc:654] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.917124 1155 tablet_bootstrap.cc:492] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.917235 1155 ts_tablet_manager.cc:1397] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.247s user 0.032s sys 0.078s
I20250629 01:51:57.917785 1155 raft_consensus.cc:357] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.917934 1155 raft_consensus.cc:383] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.917990 1155 raft_consensus.cc:738] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.918154 1155 consensus_queue.cc:260] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.918375 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0047c9ac32fc40b1a88044c935d49e94 (DEFAULT_TABLE table=default.Table_7e373110e0024c6dbab7e7b4039555a2 [id=ec8f383fbb5145099054152945f117da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.918584 1155 ts_tablet_manager.cc:1428] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:57.920320 749 tablet_service.cc:1468] Processing CreateTablet for tablet 50f2d76bc0a041c79b058c537150b894 (DEFAULT_TABLE table=default.Table_d96f54aeed554f3bbe79066de62debc5 [id=7a82929ed99342f599aff24655e5e704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.921530 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f2d76bc0a041c79b058c537150b894. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.924719 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdebad5cfed047a2ad4f58bdef3c105f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.926077 1155 tablet_bootstrap.cc:492] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:57.926535 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f2336702c5e941ce994df6edb1970192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2336702c5e941ce994df6edb1970192 already in progress: creating tablet
W20250629 01:51:57.926795 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 45461dc364d847d0b0272ac714559008 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45461dc364d847d0b0272ac714559008 already in progress: creating tablet
I20250629 01:51:57.927943 884 tablet_service.cc:1468] Processing CreateTablet for tablet 34a72207d9134115b98a334424974cc8 (DEFAULT_TABLE table=default.Table_f2fa469b443b4b3e8f3e2d272e328a4a [id=ca10890edfa242cb968c7096443f537b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.927865 744 tablet_service.cc:1468] Processing CreateTablet for tablet 0047c9ac32fc40b1a88044c935d49e94 (DEFAULT_TABLE table=default.Table_7e373110e0024c6dbab7e7b4039555a2 [id=ec8f383fbb5145099054152945f117da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.928308 747 tablet_service.cc:1468] Processing CreateTablet for tablet 34a72207d9134115b98a334424974cc8 (DEFAULT_TABLE table=default.Table_f2fa469b443b4b3e8f3e2d272e328a4a [id=ca10890edfa242cb968c7096443f537b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.928723 886 tablet_service.cc:1468] Processing CreateTablet for tablet 60f427fa19be4b3381ed9ae951cb842a (DEFAULT_TABLE table=default.Table_0c1c7d79b0a74e8ea1c1b1f5036283dd [id=eaf93619e2194a9c9fdd9670e0aaa7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.928911 1155 tablet_bootstrap.cc:654] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.932057 748 tablet_service.cc:1468] Processing CreateTablet for tablet 60f427fa19be4b3381ed9ae951cb842a (DEFAULT_TABLE table=default.Table_0c1c7d79b0a74e8ea1c1b1f5036283dd [id=eaf93619e2194a9c9fdd9670e0aaa7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.937049 1159 tablet_bootstrap.cc:492] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:57.937112 2337 raft_consensus.cc:491] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.937216 2337 raft_consensus.cc:513] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.937721 2337 leader_election.cc:290] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.938128 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4b7d0a44bbe4d8c81e8d0c444dbb55b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:57.938644 712 leader_election.cc:343] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.938818 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a72207d9134115b98a334424974cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.940397 756 tablet_service.cc:1468] Processing CreateTablet for tablet de5cf94991af4680ad3efce01b21d94f (DEFAULT_TABLE table=default.Table_a8a689877541432b88889c5998bc53fe [id=dbc27ae9473e410f853c6fe7a6639899]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.941349 1159 tablet_bootstrap.cc:654] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.942019 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0047c9ac32fc40b1a88044c935d49e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.942766 751 tablet_service.cc:1468] Processing CreateTablet for tablet fa8bc85c31eb4595905493c113fe145c (DEFAULT_TABLE table=default.Table_b4fbc288180d41e2be6faef2985f4b0e [id=6bd7ccc3994f4dd7b11462e6c86ede25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.945014 913 raft_consensus.cc:3058] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:57.947430 2363 raft_consensus.cc:491] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.947522 745 tablet_service.cc:1468] Processing CreateTablet for tablet b259bf76f013486793f3f2d00cbd1c52 (DEFAULT_TABLE table=default.Table_1d82115081f94493a4e91b0317d43803 [id=214df2ed07264a95a08a4e4b27ea897d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.947544 2363 raft_consensus.cc:513] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.948151 2363 leader_election.cc:290] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.951128 742 tablet_service.cc:1468] Processing CreateTablet for tablet 6066d69a7d9d4e8fa34f70263802d36f (DEFAULT_TABLE table=default.Table_727fa08e7ff04debbc63abc9780d6c21 [id=e9927e09acf341d19d6c73cbaa883009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.953059 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f427fa19be4b3381ed9ae951cb842a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.954720 1159 tablet_bootstrap.cc:492] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:57.955695 746 tablet_service.cc:1468] Processing CreateTablet for tablet c4790e1c22e946fe81f696943279c162 (DEFAULT_TABLE table=default.Table_af755a1cc02841f2a5ec1585c4d9d386 [id=9247182f73f84a2fb7a5f64a104174bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.955848 1159 ts_tablet_manager.cc:1397] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20250629 01:51:57.957492 1155 tablet_bootstrap.cc:492] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:57.957585 1155 ts_tablet_manager.cc:1397] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.004s
I20250629 01:51:57.958097 752 tablet_service.cc:1468] Processing CreateTablet for tablet 104769ae36604b17b948c10a1fd48350 (DEFAULT_TABLE table=default.Table_978c09358b7649f1b04db780f28564c8 [id=0f86cfece4344122afdb88a13e8b215a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.958149 1155 raft_consensus.cc:357] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.958287 1155 raft_consensus.cc:383] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.958326 1155 raft_consensus.cc:738] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.958381 2374 raft_consensus.cc:491] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.958472 1155 consensus_queue.cc:260] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.958478 2374 raft_consensus.cc:513] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.958882 1155 ts_tablet_manager.cc:1428] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:57.958937 2374 leader_election.cc:290] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.959571 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819ed44d98f14075b9a67b88dc44e86e" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:57.959856 916 raft_consensus.cc:3058] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:57.960903 986 leader_election.cc:343] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.962558 2433 raft_consensus.cc:491] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:57.962662 2433 raft_consensus.cc:513] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.963147 2433 leader_election.cc:290] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:57.962364 1159 raft_consensus.cc:357] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.963379 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cedbd093b4420d8f7b29219ce8f77b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.963617 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819ed44d98f14075b9a67b88dc44e86e" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.963793 884 tablet_service.cc:1468] Processing CreateTablet for tablet fa8bc85c31eb4595905493c113fe145c (DEFAULT_TABLE table=default.Table_b4fbc288180d41e2be6faef2985f4b0e [id=6bd7ccc3994f4dd7b11462e6c86ede25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.963797 878 tablet_service.cc:1468] Processing CreateTablet for tablet b259bf76f013486793f3f2d00cbd1c52 (DEFAULT_TABLE table=default.Table_1d82115081f94493a4e91b0317d43803 [id=214df2ed07264a95a08a4e4b27ea897d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.964155 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b259bf76f013486793f3f2d00cbd1c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.964123 895 tablet_service.cc:1468] Processing CreateTablet for tablet 104769ae36604b17b948c10a1fd48350 (DEFAULT_TABLE table=default.Table_978c09358b7649f1b04db780f28564c8 [id=0f86cfece4344122afdb88a13e8b215a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.964289 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9cfff25bea764e62954e49611b71703c" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:57.964327 896 tablet_service.cc:1468] Processing CreateTablet for tablet c4790e1c22e946fe81f696943279c162 (DEFAULT_TABLE table=default.Table_af755a1cc02841f2a5ec1585c4d9d386 [id=9247182f73f84a2fb7a5f64a104174bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.964403 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104769ae36604b17b948c10a1fd48350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.964591 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4790e1c22e946fe81f696943279c162. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:57.964838 712 leader_election.cc:343] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.967031 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8bc85c31eb4595905493c113fe145c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.967275 1155 tablet_bootstrap.cc:492] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:57.969488 740 tablet_service.cc:1468] Processing CreateTablet for tablet 9919489a17c14ed8a92b4518cf5b14f7 (DEFAULT_TABLE table=default.Table_151fdf3b919844009a4deed7768b613b [id=6fbcf393ad67415d86e2c1afc131be9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.969743 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9919489a17c14ed8a92b4518cf5b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.970145 1159 raft_consensus.cc:383] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:57.970618 1159 raft_consensus.cc:738] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:57.971227 1155 tablet_bootstrap.cc:654] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.971455 755 tablet_service.cc:1468] Processing CreateTablet for tablet 678b3a67f3cf45a584bac4a5e210f8ff (DEFAULT_TABLE table=default.Table_03d275243cf84467ab2396313192c17c [id=974d4e9832094baea54983eb8122415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.971822 883 tablet_service.cc:1468] Processing CreateTablet for tablet 678b3a67f3cf45a584bac4a5e210f8ff (DEFAULT_TABLE table=default.Table_03d275243cf84467ab2396313192c17c [id=974d4e9832094baea54983eb8122415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.972086 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678b3a67f3cf45a584bac4a5e210f8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.972575 888 tablet_service.cc:1468] Processing CreateTablet for tablet 9919489a17c14ed8a92b4518cf5b14f7 (DEFAULT_TABLE table=default.Table_151fdf3b919844009a4deed7768b613b [id=6fbcf393ad67415d86e2c1afc131be9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.972847 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9919489a17c14ed8a92b4518cf5b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.972517 885 tablet_service.cc:1468] Processing CreateTablet for tablet 6066d69a7d9d4e8fa34f70263802d36f (DEFAULT_TABLE table=default.Table_727fa08e7ff04debbc63abc9780d6c21 [id=e9927e09acf341d19d6c73cbaa883009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.972215 1159 consensus_queue.cc:260] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:57.975399 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6066d69a7d9d4e8fa34f70263802d36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.975729 889 tablet_service.cc:1468] Processing CreateTablet for tablet c30988cd89c544f7a4105dd72fe2df29 (DEFAULT_TABLE table=default.Table_5e61dd17877340b49c38eafe0498397a [id=fa1714684321459ab168b447c7e00ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.975971 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30988cd89c544f7a4105dd72fe2df29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.978636 757 tablet_service.cc:1468] Processing CreateTablet for tablet c30988cd89c544f7a4105dd72fe2df29 (DEFAULT_TABLE table=default.Table_5e61dd17877340b49c38eafe0498397a [id=fa1714684321459ab168b447c7e00ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.979116 879 tablet_service.cc:1468] Processing CreateTablet for tablet 743a4645af6c43e4a94ab7554da06521 (DEFAULT_TABLE table=default.Table_e64a7cfad754440186ceb2c9b3fc538e [id=5750431cab9a4a4c95ee16ec52f2ea1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.979382 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 743a4645af6c43e4a94ab7554da06521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.980233 750 tablet_service.cc:1468] Processing CreateTablet for tablet 743a4645af6c43e4a94ab7554da06521 (DEFAULT_TABLE table=default.Table_e64a7cfad754440186ceb2c9b3fc538e [id=5750431cab9a4a4c95ee16ec52f2ea1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.981768 1159 ts_tablet_manager.cc:1428] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:51:57.982757 1159 tablet_bootstrap.cc:492] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:57.984226 988 leader_election.cc:343] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:57.984381 988 leader_election.cc:343] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:57.984439 988 leader_election.cc:304] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:57.985087 2374 raft_consensus.cc:2747] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:57.986809 743 tablet_service.cc:1468] Processing CreateTablet for tablet 16c544420ee242b691671da85406e766 (DEFAULT_TABLE table=default.Table_e3274fb78ed8400d9bc8be767b2e174c [id=40879c0375ce440283b63a93ef41a2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.987564 1156 ts_tablet_manager.cc:1918] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:57.987651 1156 log.cc:1199] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:57.987903 1156 ts_tablet_manager.cc:1939] T 54f38ccc485043ba8b8a6f644a3e4f36 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:57.988289 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b259bf76f013486793f3f2d00cbd1c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.988667 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4790e1c22e946fe81f696943279c162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.988986 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104769ae36604b17b948c10a1fd48350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.989300 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678b3a67f3cf45a584bac4a5e210f8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.989563 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 743a4645af6c43e4a94ab7554da06521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.990152 892 tablet_service.cc:1468] Processing CreateTablet for tablet 16c544420ee242b691671da85406e766 (DEFAULT_TABLE table=default.Table_e3274fb78ed8400d9bc8be767b2e174c [id=40879c0375ce440283b63a93ef41a2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.990895 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0047c9ac32fc40b1a88044c935d49e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.991109 1159 tablet_bootstrap.cc:654] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:57.991282 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f427fa19be4b3381ed9ae951cb842a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.991613 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5cf94991af4680ad3efce01b21d94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.991978 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8bc85c31eb4595905493c113fe145c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.992339 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6066d69a7d9d4e8fa34f70263802d36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.992888 893 tablet_service.cc:1468] Processing CreateTablet for tablet 604ca49cd0f843489cd9d57784259199 (DEFAULT_TABLE table=default.Table_84d666c949fc4d9399e1895cc6ed684a [id=5723498ce6054ffda0477f15fb42aa3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.993104 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c544420ee242b691671da85406e766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.993582 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30988cd89c544f7a4105dd72fe2df29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.994050 891 tablet_service.cc:1468] Processing CreateTablet for tablet 504c15b3b168499d9a82584f82a0b792 (DEFAULT_TABLE table=default.Table_3f36a3b189db4f4296fd81d47125b49c [id=d536fd8b154449759b12f53a6ec7970f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:57.994369 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a72207d9134115b98a334424974cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:57.995523 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c544420ee242b691671da85406e766. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:57.998597 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 548674f1507743228fcd04bfd91500ae (table default.Table_89c922281dde4b02ae3a2875c9ff73ef [id=c5b98260cdb64b92964547b6d11f2f6b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 548674f1507743228fcd04bfd91500ae consensus state has no leader
I20250629 01:51:57.998808 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:58.001408 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504c15b3b168499d9a82584f82a0b792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.004355 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604ca49cd0f843489cd9d57784259199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.005808 1159 tablet_bootstrap.cc:492] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.006314 1159 ts_tablet_manager.cc:1397] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20250629 01:51:58.010262 1159 raft_consensus.cc:357] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.011128 1159 raft_consensus.cc:383] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.011397 1159 raft_consensus.cc:738] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.012382 1159 consensus_queue.cc:260] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.023229 753 tablet_service.cc:1468] Processing CreateTablet for tablet 604ca49cd0f843489cd9d57784259199 (DEFAULT_TABLE table=default.Table_84d666c949fc4d9399e1895cc6ed684a [id=5723498ce6054ffda0477f15fb42aa3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.034632 1155 tablet_bootstrap.cc:492] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.034803 1155 ts_tablet_manager.cc:1397] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.001s
I20250629 01:51:58.035530 1155 raft_consensus.cc:357] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.035748 1155 raft_consensus.cc:383] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.035842 1155 raft_consensus.cc:738] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.036103 1155 consensus_queue.cc:260] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.036612 1155 ts_tablet_manager.cc:1428] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:58.039363 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cedbd093b4420d8f7b29219ce8f77b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.041951 775 raft_consensus.cc:2466] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:58.042466 986 leader_election.cc:304] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:58.042912 2374 raft_consensus.cc:2802] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:58.042982 2374 raft_consensus.cc:491] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:58.043035 2374 raft_consensus.cc:3058] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.044456 2363 raft_consensus.cc:491] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.044553 2363 raft_consensus.cc:513] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.044993 2363 leader_election.cc:290] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.045516 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbcae02d893f4795a12d26af57b18b85" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.045676 773 raft_consensus.cc:2466] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:58.047698 986 leader_election.cc:304] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:58.048164 2363 raft_consensus.cc:2802] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:58.048233 2363 raft_consensus.cc:491] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:58.048293 2363 raft_consensus.cc:3058] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.050422 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbcae02d893f4795a12d26af57b18b85" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.050935 988 leader_election.cc:343] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.057354 877 tablet_service.cc:1468] Processing CreateTablet for tablet de5cf94991af4680ad3efce01b21d94f (DEFAULT_TABLE table=default.Table_a8a689877541432b88889c5998bc53fe [id=dbc27ae9473e410f853c6fe7a6639899]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.060315 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604ca49cd0f843489cd9d57784259199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.064529 740 tablet_service.cc:1468] Processing CreateTablet for tablet 504c15b3b168499d9a82584f82a0b792 (DEFAULT_TABLE table=default.Table_3f36a3b189db4f4296fd81d47125b49c [id=d536fd8b154449759b12f53a6ec7970f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.059257 1159 ts_tablet_manager.cc:1428] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.053s user 0.005s sys 0.009s
I20250629 01:51:58.064961 746 tablet_service.cc:1468] Processing CreateTablet for tablet f682b52dce6a4a318db9ef05431197b4 (DEFAULT_TABLE table=default.Table_93dfa58f836843938471c38606d8e6b0 [id=9584ca8a4c9c4ee3b50fec49318fa9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.065080 1159 tablet_bootstrap.cc:492] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.066107 880 tablet_service.cc:1468] Processing CreateTablet for tablet f93f07e50b414db29c081760ce2004c1 (DEFAULT_TABLE table=default.Table_001859f907034195a0b9358f6fff60c7 [id=e35befab8ebd49c490b8da54302af249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.066385 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93f07e50b414db29c081760ce2004c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.066708 1159 tablet_bootstrap.cc:654] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.076272 1159 tablet_bootstrap.cc:492] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.076839 1159 ts_tablet_manager.cc:1397] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20250629 01:51:58.077692 753 tablet_service.cc:1468] Processing CreateTablet for tablet f93f07e50b414db29c081760ce2004c1 (DEFAULT_TABLE table=default.Table_001859f907034195a0b9358f6fff60c7 [id=e35befab8ebd49c490b8da54302af249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.078001 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93f07e50b414db29c081760ce2004c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.079820 1159 raft_consensus.cc:357] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.080540 1159 raft_consensus.cc:383] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.080865 1159 raft_consensus.cc:738] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.081763 1159 consensus_queue.cc:260] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.083252 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504c15b3b168499d9a82584f82a0b792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.084079 1159 ts_tablet_manager.cc:1428] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250629 01:51:58.084316 2374 raft_consensus.cc:513] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.085032 2374 leader_election.cc:290] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.085325 1159 tablet_bootstrap.cc:492] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.086642 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cedbd093b4420d8f7b29219ce8f77b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:58.086792 773 raft_consensus.cc:3058] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.089635 757 tablet_service.cc:1468] Processing CreateTablet for tablet 8228d8449345473a85d9528913978e6f (DEFAULT_TABLE table=default.Table_b5e3cea3134a4dab8aa3e9b75293d5de [id=df0b92af23a942d5b0d9d723dd9b556b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.090962 2374 raft_consensus.cc:491] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.091073 2374 raft_consensus.cc:513] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.091531 1159 tablet_bootstrap.cc:654] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.091643 2374 leader_election.cc:290] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.092058 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40592f11f430469f9c8e82eba25b38ed" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:58.092469 986 leader_election.cc:343] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.096151 2458 raft_consensus.cc:491] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.096323 2458 raft_consensus.cc:513] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.096712 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68cedbd093b4420d8f7b29219ce8f77b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:58.096920 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40592f11f430469f9c8e82eba25b38ed" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.097543 880 tablet_service.cc:1468] Processing CreateTablet for tablet 8228d8449345473a85d9528913978e6f (DEFAULT_TABLE table=default.Table_b5e3cea3134a4dab8aa3e9b75293d5de [id=df0b92af23a942d5b0d9d723dd9b556b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.097626 745 tablet_service.cc:1468] Processing CreateTablet for tablet e4ff12d702bf48039d63c437079979fd (DEFAULT_TABLE table=default.Table_fc9218fdbd474925bd2444d014e3ba06 [id=f51d779a23d84260b34245d2a9634e72]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:58.097994 988 consensus_peers.cc:489] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:51:58.098215 988 leader_election.cc:343] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:58.098443 988 leader_election.cc:343] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.098485 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b94f8250a143b096703f2b0386e3cd" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.098558 988 leader_election.cc:304] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.098979 2374 raft_consensus.cc:2747] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.099187 751 tablet_service.cc:1468] Processing CreateTablet for tablet 573b329d020549a08c94891916164aeb (DEFAULT_TABLE table=default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:58.099591 986 leader_election.cc:343] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.100176 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b94f8250a143b096703f2b0386e3cd" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.103358 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f682b52dce6a4a318db9ef05431197b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:58.100553 988 leader_election.cc:343] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.104079 988 leader_election.cc:304] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.104329 2374 raft_consensus.cc:2747] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.104566 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 573b329d020549a08c94891916164aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.104719 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8228d8449345473a85d9528913978e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.100339 2458 leader_election.cc:290] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.106544 1159 tablet_bootstrap.cc:492] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.106879 750 tablet_service.cc:1468] Processing CreateTablet for tablet a4ec068e20d6427ebe62d8e8c0c69351 (DEFAULT_TABLE table=default.Table_02254766c6574c0c991e2621e941a252 [id=7af66798416a42b8b20a1c6c13db7699]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.107023 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ff12d702bf48039d63c437079979fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.107340 1159 ts_tablet_manager.cc:1397] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250629 01:51:58.109874 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ec068e20d6427ebe62d8e8c0c69351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.110145 1159 raft_consensus.cc:357] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.110982 1159 raft_consensus.cc:383] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.111258 1159 raft_consensus.cc:738] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.111510 885 tablet_service.cc:1468] Processing CreateTablet for tablet 3e34476c07d64a94ba8557c917b98a16 (DEFAULT_TABLE table=default.Table_a4efa38905d0481aaccc078ea261012d [id=a82ddbc470ec41ed922c1c9fa13f158e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.112028 881 tablet_service.cc:1468] Processing CreateTablet for tablet e4ff12d702bf48039d63c437079979fd (DEFAULT_TABLE table=default.Table_fc9218fdbd474925bd2444d014e3ba06 [id=f51d779a23d84260b34245d2a9634e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.112246 1159 consensus_queue.cc:260] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.112915 893 tablet_service.cc:1468] Processing CreateTablet for tablet a4ec068e20d6427ebe62d8e8c0c69351 (DEFAULT_TABLE table=default.Table_02254766c6574c0c991e2621e941a252 [id=7af66798416a42b8b20a1c6c13db7699]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.113435 1155 tablet_bootstrap.cc:492] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.114105 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f2d76bc0a041c79b058c537150b894. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.115492 1155 tablet_bootstrap.cc:654] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.115881 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1148) took 1842 ms. Trace:
I20250629 01:51:58.115996 1156 rpcz_store.cc:276] 0629 01:51:56.273415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.298259 (+ 24844us) service_pool.cc:225] Handling call
0629 01:51:57.886353 (+1588094us) ts_tablet_manager.cc:1145] Deleting tablet 54f38ccc485043ba8b8a6f644a3e4f36
0629 01:51:57.886652 (+ 299us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:57.968146 (+ 81494us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:57.987490 (+ 19344us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.115861 (+128371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587708}
I20250629 01:51:58.117262 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 54f38ccc485043ba8b8a6f644a3e4f36 (table default.Table_119622f658584820a8fcd60b2b640e05 [id=c2c6ad07071540d09fb8e8e13d4e5bea]) successfully deleted
I20250629 01:51:58.118036 755 tablet_service.cc:1468] Processing CreateTablet for tablet 3e34476c07d64a94ba8557c917b98a16 (DEFAULT_TABLE table=default.Table_a4efa38905d0481aaccc078ea261012d [id=a82ddbc470ec41ed922c1c9fa13f158e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.118405 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1151) took 1667 ms. Trace:
I20250629 01:51:58.118475 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e34476c07d64a94ba8557c917b98a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.118582 1156 rpcz_store.cc:276] 0629 01:51:56.450639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.450712 (+ 73us) service_pool.cc:225] Handling call
0629 01:51:58.116546 (+1665834us) ts_tablet_manager.cc:1145] Deleting tablet 8ad618a93c184b2fbee23eb637589e4a
0629 01:51:58.118212 (+ 1666us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.118386 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22272,"tablet-delete.queue_time_us":1665568}
I20250629 01:51:58.119310 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1152) took 1667 ms. Trace:
I20250629 01:51:58.119453 1156 rpcz_store.cc:276] 0629 01:51:56.451652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.451728 (+ 76us) service_pool.cc:225] Handling call
0629 01:51:58.119176 (+1667448us) ts_tablet_manager.cc:1145] Deleting tablet 65ab6394c32640bbb7c4cd8b77686f32
0629 01:51:58.119184 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.119296 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667186}
I20250629 01:51:58.119989 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1153) took 1609 ms. Trace:
I20250629 01:51:58.120220 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e34476c07d64a94ba8557c917b98a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.120376 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ec068e20d6427ebe62d8e8c0c69351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.124245 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8228d8449345473a85d9528913978e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.124481 1159 ts_tablet_manager.cc:1428] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
I20250629 01:51:58.125727 1159 tablet_bootstrap.cc:492] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:58.128423 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8ad618a93c184b2fbee23eb637589e4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ad618a93c184b2fbee23eb637589e4a already in progress: creating tablet
W20250629 01:51:58.128814 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 65ab6394c32640bbb7c4cd8b77686f32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65ab6394c32640bbb7c4cd8b77686f32 already in progress: creating tablet
I20250629 01:51:58.131769 1159 tablet_bootstrap.cc:654] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.111104 896 tablet_service.cc:1468] Processing CreateTablet for tablet 573b329d020549a08c94891916164aeb (DEFAULT_TABLE table=default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.135958 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 573b329d020549a08c94891916164aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.137771 1156 rpcz_store.cc:276] 0629 01:51:56.510763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.510851 (+ 88us) service_pool.cc:225] Handling call
0629 01:51:58.119891 (+1609040us) ts_tablet_manager.cc:1145] Deleting tablet eb376259f6cd490b87b52cfc1f8e6275
0629 01:51:58.119898 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.119977 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608452}
I20250629 01:51:58.138252 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1154) took 1600 ms. Trace:
I20250629 01:51:58.138362 1156 rpcz_store.cc:276] 0629 01:51:56.538111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.538191 (+ 80us) service_pool.cc:225] Handling call
0629 01:51:58.138152 (+1599961us) ts_tablet_manager.cc:1145] Deleting tablet 5a1604552122400c8fb0aca541834525
0629 01:51:58.138159 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.138239 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1599773}
I20250629 01:51:58.138813 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1156) took 1516 ms. Trace:
W20250629 01:51:58.139034 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet eb376259f6cd490b87b52cfc1f8e6275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb376259f6cd490b87b52cfc1f8e6275 already in progress: creating tablet
W20250629 01:51:58.139312 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 5a1604552122400c8fb0aca541834525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1604552122400c8fb0aca541834525 already in progress: creating tablet
I20250629 01:51:58.139546 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ff12d702bf48039d63c437079979fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.064033 890 tablet_service.cc:1468] Processing CreateTablet for tablet f682b52dce6a4a318db9ef05431197b4 (DEFAULT_TABLE table=default.Table_93dfa58f836843938471c38606d8e6b0 [id=9584ca8a4c9c4ee3b50fec49318fa9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.140925 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f682b52dce6a4a318db9ef05431197b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.142846 1159 tablet_bootstrap.cc:492] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.143332 1159 ts_tablet_manager.cc:1397] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20250629 01:51:58.144973 2456 raft_consensus.cc:491] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.145087 2456 raft_consensus.cc:513] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.145603 2456 leader_election.cc:290] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:58.145694 1156 rpcz_store.cc:276] 0629 01:51:56.621921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.621970 (+ 49us) service_pool.cc:225] Handling call
0629 01:51:58.138722 (+1516752us) ts_tablet_manager.cc:1145] Deleting tablet ea62c05f0ede4db9b48f76347e23a5cb
0629 01:51:58.138737 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.138799 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1516600}
W20250629 01:51:58.146042 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ea62c05f0ede4db9b48f76347e23a5cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea62c05f0ede4db9b48f76347e23a5cb already in progress: creating tablet
I20250629 01:51:58.137542 1155 tablet_bootstrap.cc:492] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.146242 1155 ts_tablet_manager.cc:1397] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.005s
I20250629 01:51:58.146113 1159 raft_consensus.cc:357] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.146988 1155 raft_consensus.cc:357] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.147138 1155 raft_consensus.cc:383] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.147212 1155 raft_consensus.cc:738] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.146929 1159 raft_consensus.cc:383] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.147384 1155 consensus_queue.cc:260] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:58.147441 850 leader_election.cc:343] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:58.147441 1159 raft_consensus.cc:738] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.147825 1155 ts_tablet_manager.cc:1428] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250629 01:51:58.147851 850 leader_election.cc:343] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:58.147982 850 leader_election.cc:304] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.148157 1155 tablet_bootstrap.cc:492] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.148228 2456 raft_consensus.cc:2747] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.148389 1159 consensus_queue.cc:260] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.150609 1159 ts_tablet_manager.cc:1428] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:58.151727 1159 tablet_bootstrap.cc:492] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.155253 2456 raft_consensus.cc:491] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.155354 2456 raft_consensus.cc:513] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.155753 2456 leader_election.cc:290] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:58.160907 2363 raft_consensus.cc:513] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.162053 2363 leader_election.cc:290] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.165396 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.167510 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbcae02d893f4795a12d26af57b18b85" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:58.167842 2456 raft_consensus.cc:491] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.167968 2456 raft_consensus.cc:513] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:58.168007 988 leader_election.cc:343] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.168478 2456 leader_election.cc:290] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:58.168711 2456 raft_consensus.cc:491] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.168804 2456 raft_consensus.cc:513] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.168877 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.169059 1050 raft_consensus.cc:2391] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 443c4bd3c5d548ee8cd347d2b4de5293 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:51:58.169282 2456 leader_election.cc:290] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:58.169628 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.172569 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Bootstrap replayed 1/1 log segments. Stats: ops{read=761 overwritten=0 applied=761 ignored=0} inserts{seen=686 ignored=0} mutations{seen=868 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250629 01:51:58.151526 773 raft_consensus.cc:2466] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:58.161805 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbcae02d893f4795a12d26af57b18b85" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:58.174630 775 raft_consensus.cc:3058] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.174834 986 leader_election.cc:304] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:58.175176 2363 raft_consensus.cc:2802] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:58.175659 2363 raft_consensus.cc:695] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:58.175949 2068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: Bootstrap complete.
I20250629 01:51:58.176031 2363 consensus_queue.cc:237] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.178047 1155 tablet_bootstrap.cc:654] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.165699 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "462789e8daef42a08ff8b2cd2ddd1c49" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.167318 2337 raft_consensus.cc:491] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.184900 2337 raft_consensus.cc:513] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:58.185118 712 consensus_peers.cc:489] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:58.185312 850 leader_election.cc:304] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.185405 2337 leader_election.cc:290] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.169283 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95247cc778504f4d85d9013f84c59e69" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.170002 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8bb4d7c2096a45df8eb392a2e7d85c3d" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.185582 777 raft_consensus.cc:2391] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 443c4bd3c5d548ee8cd347d2b4de5293 in current term 1: Already voted for candidate 4712643b602e4f898752fcab6a1e5913 in this term.
I20250629 01:51:58.185923 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.186096 2456 raft_consensus.cc:2747] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:51:58.186414 712 leader_election.cc:343] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.186725 850 leader_election.cc:304] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.186921 2456 raft_consensus.cc:3058] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.187911 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5cf94991af4680ad3efce01b21d94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.189123 850 leader_election.cc:304] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.189392 2454 raft_consensus.cc:3058] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.163143 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1157) took 1536 ms. Trace:
I20250629 01:51:58.190004 1156 rpcz_store.cc:276] 0629 01:51:56.626623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.626697 (+ 74us) service_pool.cc:225] Handling call
0629 01:51:58.163055 (+1536358us) ts_tablet_manager.cc:1145] Deleting tablet f4a3b7d971694e15be1aaa072468dae2
0629 01:51:58.163063 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.163130 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536215}
W20250629 01:51:58.190471 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f4a3b7d971694e15be1aaa072468dae2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f4a3b7d971694e15be1aaa072468dae2 already in progress: creating tablet
I20250629 01:51:58.190655 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1158) took 1543 ms. Trace:
I20250629 01:51:58.190763 1156 rpcz_store.cc:276] 0629 01:51:56.647527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.647573 (+ 46us) service_pool.cc:225] Handling call
0629 01:51:58.190516 (+1542943us) ts_tablet_manager.cc:1145] Deleting tablet 898b49812ece4b57a1a1015d9f26be2f
0629 01:51:58.190523 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.190642 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542797}
W20250629 01:51:58.191164 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 898b49812ece4b57a1a1015d9f26be2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898b49812ece4b57a1a1015d9f26be2f already in progress: creating tablet
I20250629 01:51:58.191247 1156 tablet_replica.cc:331] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.191504 1156 raft_consensus.cc:2241] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.191695 1156 raft_consensus.cc:2270] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.192384 1156 ts_tablet_manager.cc:1905] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.195497 2068 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:58.196518 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1ece6af955a54ec2b5819ceba5d2ec17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:58.197055 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ece6af955a54ec2b5819ceba5d2ec17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:58.197158 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet e5e8b8e4d42f43c78f34102b76b4b11a (table default.Table_904b8b7a3ab1414a9bd29274648bbb86 [id=e991e29b3d9d4dcc96912754565336f8]). Expected version 1 got 0
I20250629 01:51:58.201828 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:51:58.202095 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ec0eeb0ef02c416abc5665f058c6bd7f
I20250629 01:51:58.202631 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0eeb0ef02c416abc5665f058c6bd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:58.204638 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:51:58.206040 1159 tablet_bootstrap.cc:654] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.207355 1155 tablet_bootstrap.cc:492] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.207507 1155 ts_tablet_manager.cc:1397] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250629 01:51:58.208456 1155 raft_consensus.cc:357] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.208701 2068 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4258c004e04bcf8216bb7c8fd56c54, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.209470 2068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 761, Last appended: 1.761, Last appended by leader: 761, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:58.209697 1155 raft_consensus.cc:383] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.209780 1155 raft_consensus.cc:738] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.209971 1155 consensus_queue.cc:260] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.210713 1155 ts_tablet_manager.cc:1428] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:58.210965 1155 tablet_bootstrap.cc:492] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:51:58.214718 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4ff12d702bf48039d63c437079979fd consensus state has no leader
I20250629 01:51:58.214841 1155 tablet_bootstrap.cc:654] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.216346 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2a876b3db3904a4297b5b13a9325b1c3
I20250629 01:51:58.217092 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a876b3db3904a4297b5b13a9325b1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:58.222891 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a876b3db3904a4297b5b13a9325b1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:58.223765 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2a876b3db3904a4297b5b13a9325b1c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a876b3db3904a4297b5b13a9325b1c3 already in progress: creating tablet
I20250629 01:51:58.224476 2433 raft_consensus.cc:491] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.224606 2433 raft_consensus.cc:513] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.225191 2433 leader_election.cc:290] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.225710 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74b3badd684d45fa88dc8cbcfc61001a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.226147 712 leader_election.cc:343] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.230209 1155 tablet_bootstrap.cc:492] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.230440 1159 tablet_bootstrap.cc:492] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.231165 1155 ts_tablet_manager.cc:1397] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250629 01:51:58.232343 1155 raft_consensus.cc:357] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.233103 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:58.233477 2458 consensus_queue.cc:1035] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:58.234280 896 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0eeb0ef02c416abc5665f058c6bd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:58.235877 1032 tablet_service.cc:1468] Processing CreateTablet for tablet cdebad5cfed047a2ad4f58bdef3c105f (DEFAULT_TABLE table=default.Table_bac468da747c48e4bf6c7884bbd9e1f1 [id=5fe17a5d54304ae5ae72c6cde593538e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.237644 2456 raft_consensus.cc:2747] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.239940 2454 raft_consensus.cc:2747] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.243317 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:58.243453 1155 raft_consensus.cc:383] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.243542 1155 raft_consensus.cc:738] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.235013 2068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: configured and running, proceeding with master startup.
I20250629 01:51:58.244414 775 raft_consensus.cc:2466] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
W20250629 01:51:58.245185 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ec0eeb0ef02c416abc5665f058c6bd7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0eeb0ef02c416abc5665f058c6bd7f already in progress: creating tablet
I20250629 01:51:58.245198 1155 consensus_queue.cc:260] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.245512 986 leader_election.cc:304] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:58.245636 1155 ts_tablet_manager.cc:1428] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250629 01:51:58.245755 2458 raft_consensus.cc:2802] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:58.245886 2458 raft_consensus.cc:695] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:58.245982 1155 tablet_bootstrap.cc:492] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.246150 2458 consensus_queue.cc:237] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.247328 1155 tablet_bootstrap.cc:654] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.247351 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdebad5cfed047a2ad4f58bdef3c105f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:58.261461 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a43f6594e4844af580c99ea598272aae consensus state has no leader
I20250629 01:51:58.264185 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_97c5093f3a1542f5bb7fc463c1955f9e" } new_table_name: "default.Table_23b66e043cc746f68172596b5afc01c9" modify_external_catalogs: true
I20250629 01:51:58.267542 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4bca012737544ca1839fbb5ab17332b7" } new_table_name: "default.Table_e0d3931a857c4c9b86f1596c5ac6b3f3" modify_external_catalogs: true
I20250629 01:51:58.268885 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_84c1f44d16dd4a46b714a94ea498d75d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:58.269407 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84c1f44d16dd4a46b714a94ea498d75d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:58.270519 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14492ecf17b84f3f8b5fb91233a242dd" } modify_external_catalogs: true
I20250629 01:51:58.270664 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_14492ecf17b84f3f8b5fb91233a242dd" } modify_external_catalogs: true
I20250629 01:51:58.272442 1154 tablet_bootstrap.cc:492] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:58.272607 1154 ts_tablet_manager.cc:1397] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.364s user 0.006s sys 0.048s
I20250629 01:51:58.272557 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f52b1033085542258b846453f2b91f3d" } modify_external_catalogs: true
I20250629 01:51:58.272724 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f52b1033085542258b846453f2b91f3d" } modify_external_catalogs: true
I20250629 01:51:58.273298 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f9574ad3b53b407d9a3352687623467a" } new_table_name: "default.Table_22439f95e0a3488c8ffc11992703db96" modify_external_catalogs: true
I20250629 01:51:58.273352 1154 raft_consensus.cc:357] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.273912 1154 raft_consensus.cc:383] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.274019 1154 raft_consensus.cc:738] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.274210 1154 consensus_queue.cc:260] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.274909 1154 ts_tablet_manager.cc:1428] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:58.275220 1154 tablet_bootstrap.cc:492] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:58.230773 1159 ts_tablet_manager.cc:1397] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.003s
I20250629 01:51:58.280930 1159 raft_consensus.cc:357] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:58.281144 1159 raft_consensus.cc:383] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.281245 1159 raft_consensus.cc:738] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.281512 1159 consensus_queue.cc:260] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:58.282125 1159 ts_tablet_manager.cc:1428] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:51:58.282398 1159 tablet_bootstrap.cc:492] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.291522 1156 ts_tablet_manager.cc:1918] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.291941 1156 log.cc:1199] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/2845d8fbf3f34c769027b9b6bb3e1f08
I20250629 01:51:58.292341 1156 ts_tablet_manager.cc:1939] T 2845d8fbf3f34c769027b9b6bb3e1f08 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.293697 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1159) took 1617 ms. Trace:
I20250629 01:51:58.293844 1156 rpcz_store.cc:276] 0629 01:51:56.676519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.676561 (+ 42us) service_pool.cc:225] Handling call
0629 01:51:58.191186 (+1514625us) ts_tablet_manager.cc:1145] Deleting tablet 2845d8fbf3f34c769027b9b6bb3e1f08
0629 01:51:58.191194 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.254335 (+ 63141us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.291429 (+ 37094us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.293680 (+ 2251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1501078}
I20250629 01:51:58.294435 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 2845d8fbf3f34c769027b9b6bb3e1f08 (table default.Table_208f2046cc46452887e343bab242e4b2 [id=77c157c54c864ddbb4678e9af6b1fe43]) successfully deleted
I20250629 01:51:58.294903 1156 tablet_replica.cc:331] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.295609 1156 raft_consensus.cc:2241] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.296327 1156 raft_consensus.cc:2270] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.297804 1156 ts_tablet_manager.cc:1905] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.298629 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet f2336702c5e941ce994df6edb1970192 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.298952 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 45461dc364d847d0b0272ac714559008 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.299831 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 50f2d76bc0a041c79b058c537150b894 (DEFAULT_TABLE table=default.Table_d96f54aeed554f3bbe79066de62debc5 [id=7a82929ed99342f599aff24655e5e704]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.300472 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0047c9ac32fc40b1a88044c935d49e94 (DEFAULT_TABLE table=default.Table_7e373110e0024c6dbab7e7b4039555a2 [id=ec8f383fbb5145099054152945f117da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.301192 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 34a72207d9134115b98a334424974cc8 (DEFAULT_TABLE table=default.Table_f2fa469b443b4b3e8f3e2d272e328a4a [id=ca10890edfa242cb968c7096443f537b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.301550 2454 raft_consensus.cc:491] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.301666 2454 raft_consensus.cc:513] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.302263 2454 leader_election.cc:290] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:58.302716 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 60f427fa19be4b3381ed9ae951cb842a (DEFAULT_TABLE table=default.Table_0c1c7d79b0a74e8ea1c1b1f5036283dd [id=eaf93619e2194a9c9fdd9670e0aaa7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.303261 1026 tablet_service.cc:1468] Processing CreateTablet for tablet de5cf94991af4680ad3efce01b21d94f (DEFAULT_TABLE table=default.Table_a8a689877541432b88889c5998bc53fe [id=dbc27ae9473e410f853c6fe7a6639899]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.303313 1023 tablet_service.cc:1468] Processing CreateTablet for tablet fa8bc85c31eb4595905493c113fe145c (DEFAULT_TABLE table=default.Table_b4fbc288180d41e2be6faef2985f4b0e [id=6bd7ccc3994f4dd7b11462e6c86ede25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.304019 1015 tablet_service.cc:1468] Processing CreateTablet for tablet b259bf76f013486793f3f2d00cbd1c52 (DEFAULT_TABLE table=default.Table_1d82115081f94493a4e91b0317d43803 [id=214df2ed07264a95a08a4e4b27ea897d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:58.304422 850 leader_election.cc:343] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:58.304539 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 6066d69a7d9d4e8fa34f70263802d36f (DEFAULT_TABLE table=default.Table_727fa08e7ff04debbc63abc9780d6c21 [id=e9927e09acf341d19d6c73cbaa883009]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.304857 2337 raft_consensus.cc:491] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.304967 2337 raft_consensus.cc:513] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:58.305043 850 leader_election.cc:343] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:58.305133 850 leader_election.cc:304] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.305485 2337 leader_election.cc:290] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.306186 2454 raft_consensus.cc:2747] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.306592 1154 tablet_bootstrap.cc:654] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.306727 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06e31520d3ad4c3395a59049755a4725" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.307313 712 leader_election.cc:343] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.307931 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c4790e1c22e946fe81f696943279c162 (DEFAULT_TABLE table=default.Table_af755a1cc02841f2a5ec1585c4d9d386 [id=9247182f73f84a2fb7a5f64a104174bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.308075 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 678b3a67f3cf45a584bac4a5e210f8ff (DEFAULT_TABLE table=default.Table_03d275243cf84467ab2396313192c17c [id=974d4e9832094baea54983eb8122415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.308218 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 104769ae36604b17b948c10a1fd48350 (DEFAULT_TABLE table=default.Table_978c09358b7649f1b04db780f28564c8 [id=0f86cfece4344122afdb88a13e8b215a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.308549 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 9919489a17c14ed8a92b4518cf5b14f7 (DEFAULT_TABLE table=default.Table_151fdf3b919844009a4deed7768b613b [id=6fbcf393ad67415d86e2c1afc131be9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.308786 1020 tablet_service.cc:1468] Processing CreateTablet for tablet c30988cd89c544f7a4105dd72fe2df29 (DEFAULT_TABLE table=default.Table_5e61dd17877340b49c38eafe0498397a [id=fa1714684321459ab168b447c7e00ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.321360 1159 tablet_bootstrap.cc:654] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.324756 912 raft_consensus.cc:3058] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.326408 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_34d94473b0b1431ba7da4d005a8f3b91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:58.326594 2458 raft_consensus.cc:491] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.326851 2458 raft_consensus.cc:513] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:58.327023 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34d94473b0b1431ba7da4d005a8f3b91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:58.327872 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9d77381f7e4241aa546eaea3077168" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.328097 777 raft_consensus.cc:2466] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:58.328516 986 leader_election.cc:304] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:58.328718 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7a293230f8fb43b0aab93a9e76852915"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:58.329180 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a293230f8fb43b0aab93a9e76852915 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:58.329237 2458 leader_election.cc:290] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.329424 2458 raft_consensus.cc:2802] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:58.329510 2458 raft_consensus.cc:491] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:58.329600 2458 raft_consensus.cc:3058] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.332149 2458 raft_consensus.cc:513] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.333108 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9d77381f7e4241aa546eaea3077168" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:58.333290 777 raft_consensus.cc:3058] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.333974 1154 tablet_bootstrap.cc:492] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:58.334127 1154 ts_tablet_manager.cc:1397] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250629 01:51:58.334712 1154 raft_consensus.cc:357] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.334895 1154 raft_consensus.cc:383] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.334975 1154 raft_consensus.cc:738] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.335182 1154 consensus_queue.cc:260] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.335592 777 raft_consensus.cc:2466] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:58.335649 1154 ts_tablet_manager.cc:1428] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:58.335889 1154 tablet_bootstrap.cc:492] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:58.338085 986 leader_election.cc:304] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:58.338500 2374 raft_consensus.cc:2802] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:58.338601 2374 raft_consensus.cc:695] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:58.338773 2374 consensus_queue.cc:237] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.340409 2458 leader_election.cc:290] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:58.342085 712 consensus_peers.cc:489] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:58.342339 824 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:58.329301 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9d77381f7e4241aa546eaea3077168" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.346024 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac9d77381f7e4241aa546eaea3077168" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:58.346446 1154 tablet_bootstrap.cc:654] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:58.346537 988 leader_election.cc:343] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:58.347064 988 leader_election.cc:343] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.347358 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f2d76bc0a041c79b058c537150b894. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.347522 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678b3a67f3cf45a584bac4a5e210f8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.347823 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9919489a17c14ed8a92b4518cf5b14f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.347957 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a72207d9134115b98a334424974cc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.348240 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c30988cd89c544f7a4105dd72fe2df29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.348461 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104769ae36604b17b948c10a1fd48350. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.348601 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4790e1c22e946fe81f696943279c162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.348728 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8bc85c31eb4595905493c113fe145c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.348963 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de5cf94991af4680ad3efce01b21d94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.349118 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f427fa19be4b3381ed9ae951cb842a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.352742 2337 raft_consensus.cc:491] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.352842 2337 raft_consensus.cc:513] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.353312 2337 leader_election.cc:290] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.356473 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6066d69a7d9d4e8fa34f70263802d36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.359551 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0047c9ac32fc40b1a88044c935d49e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.362255 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b259bf76f013486793f3f2d00cbd1c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.375103 1156 ts_tablet_manager.cc:1918] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:51:58.375190 1156 log.cc:1199] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/20958c5e88454e129aa8efefb63b7f14
I20250629 01:51:58.375453 1156 ts_tablet_manager.cc:1939] T 20958c5e88454e129aa8efefb63b7f14 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.376034 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1165) took 1541 ms. Trace:
I20250629 01:51:58.376144 1156 rpcz_store.cc:276] 0629 01:51:56.834943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.834989 (+ 46us) service_pool.cc:225] Handling call
0629 01:51:58.294823 (+1459834us) ts_tablet_manager.cc:1145] Deleting tablet 20958c5e88454e129aa8efefb63b7f14
0629 01:51:58.294835 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.308942 (+ 14107us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.375019 (+ 66077us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.376019 (+ 1000us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1459666}
I20250629 01:51:58.376470 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1169) took 1412 ms. Trace:
I20250629 01:51:58.376524 1156 rpcz_store.cc:276] 0629 01:51:56.964280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.964333 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:58.376376 (+1412043us) ts_tablet_manager.cc:1145] Deleting tablet 33aa0a66cdbf49c5820e2b43b021f28b
0629 01:51:58.376385 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.376458 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411745}
I20250629 01:51:58.376693 1156 tablet_replica.cc:331] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.376833 1156 raft_consensus.cc:2241] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.376941 1156 raft_consensus.cc:2270] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.377350 1156 ts_tablet_manager.cc:1905] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.380692 1154 tablet_bootstrap.cc:492] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:58.380800 1154 ts_tablet_manager.cc:1397] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20250629 01:51:58.380818 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 20958c5e88454e129aa8efefb63b7f14 (table default.Table_b60d57f2a4b34062a9ed95ffd636f209 [id=228c92a41618432cb3f0986698fc74c8]) successfully deleted
W20250629 01:51:58.381070 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 33aa0a66cdbf49c5820e2b43b021f28b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33aa0a66cdbf49c5820e2b43b021f28b already in progress: creating tablet
I20250629 01:51:58.381963 1154 raft_consensus.cc:357] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.382089 1154 raft_consensus.cc:383] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.382126 1154 raft_consensus.cc:738] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.382283 1154 consensus_queue.cc:260] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.382664 1154 ts_tablet_manager.cc:1428] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:58.382860 1154 tablet_bootstrap.cc:492] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:58.383988 1154 tablet_bootstrap.cc:654] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.387267 1156 ts_tablet_manager.cc:1918] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.387353 1156 log.cc:1199] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/0d4320f615464fedbcab25137952f684
I20250629 01:51:58.387634 1156 ts_tablet_manager.cc:1939] T 0d4320f615464fedbcab25137952f684 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.390182 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1171) took 1425 ms. Trace:
I20250629 01:51:58.390256 1156 rpcz_store.cc:276] 0629 01:51:56.964750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.964789 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:58.376665 (+1411876us) ts_tablet_manager.cc:1145] Deleting tablet 0d4320f615464fedbcab25137952f684
0629 01:51:58.376670 (+ 5us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.379943 (+ 3273us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.387205 (+ 7262us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.390168 (+ 2963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411692}
I20250629 01:51:58.390697 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 0d4320f615464fedbcab25137952f684 (table default.Table_c7d1c2fe02d04423803f16f69f0d6625 [id=5a75eb0020fb4d4fb1d2066ed7d670a5]) successfully deleted
I20250629 01:51:58.392243 1156 tablet_replica.cc:331] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.392426 1156 raft_consensus.cc:2241] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.392570 1156 raft_consensus.cc:2270] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.393055 1156 ts_tablet_manager.cc:1905] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.413883 1156 ts_tablet_manager.cc:1918] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.417847 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fcead51fd9b43d0a52c4669a5e23fe5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.417893 1156 log.cc:1199] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a6a012ba891949538800581436ef5380
I20250629 01:51:58.418949 1156 ts_tablet_manager.cc:1939] T a6a012ba891949538800581436ef5380 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:58.419670 712 leader_election.cc:343] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.420579 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1170) took 1456 ms. Trace:
I20250629 01:51:58.420925 1156 rpcz_store.cc:276] 0629 01:51:56.964552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:56.965665 (+ 1113us) service_pool.cc:225] Handling call
0629 01:51:58.392200 (+1426535us) ts_tablet_manager.cc:1145] Deleting tablet a6a012ba891949538800581436ef5380
0629 01:51:58.392208 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.396425 (+ 4217us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.413803 (+ 17378us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.420561 (+ 6758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426390}
I20250629 01:51:58.421458 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a6a012ba891949538800581436ef5380 (table default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]) successfully deleted
I20250629 01:51:58.421813 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1179) took 1363 ms. Trace:
I20250629 01:51:58.422111 1156 rpcz_store.cc:276] 0629 01:51:57.058271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.058867 (+ 596us) service_pool.cc:225] Handling call
0629 01:51:58.421723 (+1362856us) ts_tablet_manager.cc:1145] Deleting tablet 6b0ef1ba46ac487ab1502c3d116814e1
0629 01:51:58.421731 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.421799 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362718}
W20250629 01:51:58.422672 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6b0ef1ba46ac487ab1502c3d116814e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b0ef1ba46ac487ab1502c3d116814e1 already in progress: creating tablet
I20250629 01:51:58.423698 1158 ts_tablet_manager.cc:1918] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.423806 1158 log.cc:1199] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/06e31520d3ad4c3395a59049755a4725
I20250629 01:51:58.423930 2418 raft_consensus.cc:491] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef5c5979bb9e475f983af7c3e9e0e019)
I20250629 01:51:58.424072 1158 ts_tablet_manager.cc:1939] T 06e31520d3ad4c3395a59049755a4725 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:58.424232 1159 tablet_bootstrap.cc:492] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.424361 1159 ts_tablet_manager.cc:1397] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.142s user 0.021s sys 0.020s
I20250629 01:51:58.425416 1158 tablet_replica.cc:331] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:58.425599 1158 raft_consensus.cc:2241] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.425715 1158 raft_consensus.cc:2270] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.425976 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1178) took 1367 ms. Trace:
I20250629 01:51:58.426153 1158 ts_tablet_manager.cc:1905] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.426476 2418 raft_consensus.cc:513] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:58.427083 2418 leader_election.cc:290] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:58.427686 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.428951 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b55c66136ba43d7b535750e04b72308" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.429739 1156 rpcz_store.cc:276] 0629 01:51:57.058158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.060738 (+ 2580us) service_pool.cc:225] Handling call
0629 01:51:58.425890 (+1365152us) ts_tablet_manager.cc:1145] Deleting tablet 962f1ac223074f11a69061a69396b3aa
0629 01:51:58.425897 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.425963 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365001}
W20250629 01:51:58.430289 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 962f1ac223074f11a69061a69396b3aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 962f1ac223074f11a69061a69396b3aa already in progress: creating tablet
I20250629 01:51:58.430831 850 leader_election.cc:304] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.431501 2418 raft_consensus.cc:2747] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:58.433840 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 743a4645af6c43e4a94ab7554da06521 (DEFAULT_TABLE table=default.Table_e64a7cfad754440186ceb2c9b3fc538e [id=5750431cab9a4a4c95ee16ec52f2ea1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.434161 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 743a4645af6c43e4a94ab7554da06521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.436365 1159 raft_consensus.cc:357] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.436553 1159 raft_consensus.cc:383] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.436636 1159 raft_consensus.cc:738] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.436837 1159 consensus_queue.cc:260] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.439406 2458 raft_consensus.cc:491] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.439504 2458 raft_consensus.cc:513] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.439952 2458 leader_election.cc:290] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.439909 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 16c544420ee242b691671da85406e766 (DEFAULT_TABLE table=default.Table_e3274fb78ed8400d9bc8be767b2e174c [id=40879c0375ce440283b63a93ef41a2a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.440222 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16c544420ee242b691671da85406e766. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.440598 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fcead51fd9b43d0a52c4669a5e23fe5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.440732 777 raft_consensus.cc:2466] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:58.441082 986 leader_election.cc:304] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:58.441334 2458 raft_consensus.cc:2802] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:58.441392 2458 raft_consensus.cc:491] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:58.441439 2458 raft_consensus.cc:3058] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.443517 2458 raft_consensus.cc:513] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.444033 2458 leader_election.cc:290] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.444484 1159 ts_tablet_manager.cc:1428] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20250629 01:51:58.444541 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fcead51fd9b43d0a52c4669a5e23fe5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:58.444655 777 raft_consensus.cc:3058] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.444752 1159 tablet_bootstrap.cc:492] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.446004 1159 tablet_bootstrap.cc:654] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.446604 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 504c15b3b168499d9a82584f82a0b792 (DEFAULT_TABLE table=default.Table_3f36a3b189db4f4296fd81d47125b49c [id=d536fd8b154449759b12f53a6ec7970f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.446902 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504c15b3b168499d9a82584f82a0b792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.446949 777 raft_consensus.cc:2466] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:58.447646 986 leader_election.cc:304] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:58.448259 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fcead51fd9b43d0a52c4669a5e23fe5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.448645 988 leader_election.cc:343] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.448911 2458 raft_consensus.cc:2802] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:58.449039 2458 raft_consensus.cc:695] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:58.449285 2458 consensus_queue.cc:237] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.450812 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fcead51fd9b43d0a52c4669a5e23fe5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:58.451411 988 leader_election.cc:343] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.452467 1031 tablet_service.cc:1468] Processing CreateTablet for tablet f93f07e50b414db29c081760ce2004c1 (DEFAULT_TABLE table=default.Table_001859f907034195a0b9358f6fff60c7 [id=e35befab8ebd49c490b8da54302af249]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.452791 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93f07e50b414db29c081760ce2004c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.458549 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e4ff12d702bf48039d63c437079979fd (DEFAULT_TABLE table=default.Table_fc9218fdbd474925bd2444d014e3ba06 [id=f51d779a23d84260b34245d2a9634e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.458896 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4ff12d702bf48039d63c437079979fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.462229 1156 tablet_replica.cc:331] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.462706 1156 raft_consensus.cc:2241] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.438838 2487 tablet.cc:1722] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250629 01:51:58.463876 1156 raft_consensus.cc:2270] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.465363 1156 ts_tablet_manager.cc:1905] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.465598 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 8228d8449345473a85d9528913978e6f (DEFAULT_TABLE table=default.Table_b5e3cea3134a4dab8aa3e9b75293d5de [id=df0b92af23a942d5b0d9d723dd9b556b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.465917 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8228d8449345473a85d9528913978e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.471896 1159 tablet_bootstrap.cc:492] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.472086 1159 ts_tablet_manager.cc:1397] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20250629 01:51:58.472898 1159 raft_consensus.cc:357] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.473134 1159 raft_consensus.cc:383] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.473232 1159 raft_consensus.cc:738] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.473500 1159 consensus_queue.cc:260] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.474294 1159 ts_tablet_manager.cc:1428] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:58.474552 1159 tablet_bootstrap.cc:492] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.475943 1159 tablet_bootstrap.cc:654] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.477545 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a4ec068e20d6427ebe62d8e8c0c69351 (DEFAULT_TABLE table=default.Table_02254766c6574c0c991e2621e941a252 [id=7af66798416a42b8b20a1c6c13db7699]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.477833 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ec068e20d6427ebe62d8e8c0c69351. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.480144 1159 tablet_bootstrap.cc:492] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.480242 1159 ts_tablet_manager.cc:1397] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250629 01:51:58.480767 1159 raft_consensus.cc:357] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.480877 1159 raft_consensus.cc:383] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.480914 1159 raft_consensus.cc:738] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.481056 1159 consensus_queue.cc:260] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.481228 2454 raft_consensus.cc:491] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.481340 2454 raft_consensus.cc:513] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:58.481477 1159 ts_tablet_manager.cc:1428] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:58.481715 1159 tablet_bootstrap.cc:492] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.481815 2454 leader_election.cc:290] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:51:58.482967 850 leader_election.cc:343] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: c0b9afb9976541b2bc7ab188f35a62ec
W20250629 01:51:58.483135 850 leader_election.cc:343] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: c0b9afb9976541b2bc7ab188f35a62ec
I20250629 01:51:58.483196 850 leader_election.cc:304] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.483379 2454 raft_consensus.cc:2747] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.483498 1159 tablet_bootstrap.cc:654] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.484670 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 573b329d020549a08c94891916164aeb (DEFAULT_TABLE table=default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.484959 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 573b329d020549a08c94891916164aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.487807 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0eeb0ef02c416abc5665f058c6bd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.488368 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6092e19acf974f2dad5a903da72944ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.488878 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9d77381f7e4241aa546eaea3077168 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.489220 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet fbcae02d893f4795a12d26af57b18b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.489564 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.489876 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.490197 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 40592f11f430469f9c8e82eba25b38ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.490618 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3e34476c07d64a94ba8557c917b98a16 (DEFAULT_TABLE table=default.Table_a4efa38905d0481aaccc078ea261012d [id=a82ddbc470ec41ed922c1c9fa13f158e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.490945 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e34476c07d64a94ba8557c917b98a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.470573 1156 ts_tablet_manager.cc:1918] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.493224 1156 log.cc:1199] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/648b0a0f742b4e7da9eaff6bd8795fc4
I20250629 01:51:58.493495 1156 ts_tablet_manager.cc:1939] T 648b0a0f742b4e7da9eaff6bd8795fc4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.494088 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1192) took 1153 ms. Trace:
I20250629 01:51:58.494168 1156 rpcz_store.cc:276] 0629 01:51:57.340619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.340815 (+ 196us) service_pool.cc:225] Handling call
0629 01:51:58.462188 (+1121373us) ts_tablet_manager.cc:1145] Deleting tablet 648b0a0f742b4e7da9eaff6bd8795fc4
0629 01:51:58.462197 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.468160 (+ 5963us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.470512 (+ 2352us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.494073 (+ 23561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1121061}
I20250629 01:51:58.494562 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 648b0a0f742b4e7da9eaff6bd8795fc4 (table default.Table_afb98d5207ae4d2d8b97b8577ae3a157 [id=e874833b32de4076921fdf0e2f103a1a]) successfully deleted
I20250629 01:51:58.495031 1155 tablet_bootstrap.cc:492] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.495203 1155 ts_tablet_manager.cc:1397] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.249s user 0.005s sys 0.033s
I20250629 01:51:58.495944 1155 raft_consensus.cc:357] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:58.497431 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1ad1b7de79b48058437a9dea5d30b37 consensus state has no leader
I20250629 01:51:58.498240 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_708d2b9c3b804e6288ce7b9226d76bac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:58.498797 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_708d2b9c3b804e6288ce7b9226d76bac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:58.501279 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c946c24832f9401d9fc8864bf45231ff consensus state has no leader
I20250629 01:51:58.501510 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 84cfa2d73c6b4ca2bf41437b196d10b1 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:58.503759 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4622cfe68f7a49a8a6d84d12a686dea7" } modify_external_catalogs: true
I20250629 01:51:58.503902 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4622cfe68f7a49a8a6d84d12a686dea7" } modify_external_catalogs: true
W20250629 01:51:58.514678 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a5c7722575e452aa009931d63637b93 consensus state has no leader
I20250629 01:51:58.515419 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6fda51d9864f43b6ae070bb2a165ebad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:58.515976 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fda51d9864f43b6ae070bb2a165ebad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:58.517480 2487 tablet.cc:1722] T 95247cc778504f4d85d9013f84c59e69 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:58.517867 1021 tablet_service.cc:1468] Processing CreateTablet for tablet f682b52dce6a4a318db9ef05431197b4 (DEFAULT_TABLE table=default.Table_93dfa58f836843938471c38606d8e6b0 [id=9584ca8a4c9c4ee3b50fec49318fa9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.518154 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f682b52dce6a4a318db9ef05431197b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.518971 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 604ca49cd0f843489cd9d57784259199 (DEFAULT_TABLE table=default.Table_84d666c949fc4d9399e1895cc6ed684a [id=5723498ce6054ffda0477f15fb42aa3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.519225 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 604ca49cd0f843489cd9d57784259199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.520680 2487 tablet.cc:1717] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Skipping requested alter to schema version 4, tablet already version 4
I20250629 01:51:58.526293 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 2a876b3db3904a4297b5b13a9325b1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:57 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:58.532752 2487 tablet.cc:1722] T 176bd3e921c74f9a93d4b2f1c0fc97a7 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:58.535104 1048 raft_consensus.cc:1273] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:58.535854 1050 raft_consensus.cc:1273] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:58.535949 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f4b7d0a44bbe4d8c81e8d0c444dbb55b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.536137 1048 raft_consensus.cc:2466] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:58.537379 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.537504 1048 raft_consensus.cc:2466] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:58.538407 1048 raft_consensus.cc:1273] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:51:58.557713 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:51:58.560720 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_3592b5e7e0c44766a622a04a3a5c9b03 [id=ed3c8fa707924f8ea53d590a7f9b3d68] alter complete (version 2)
I20250629 01:51:58.561264 2454 raft_consensus.cc:491] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.561411 2454 raft_consensus.cc:513] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.562374 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.586941 877 tablet_service.cc:1468] Processing CreateTablet for tablet 35d999aeadae450184a384ab906c4af8 (DEFAULT_TABLE table=default.Table_f216b85d57b04573a5ec1eb86e843273 [id=a1b9f65573574372a4cd857e3d83b059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.587302 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d999aeadae450184a384ab906c4af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.501929 1156 tablet_replica.cc:331] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.596165 1155 raft_consensus.cc:383] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.596227 1155 raft_consensus.cc:738] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.596330 1156 raft_consensus.cc:2241] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.596465 1156 raft_consensus.cc:2270] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.596412 1155 consensus_queue.cc:260] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.596904 1155 ts_tablet_manager.cc:1428] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.102s user 0.001s sys 0.001s
I20250629 01:51:58.564725 745 tablet_service.cc:1468] Processing CreateTablet for tablet 707ec397649f475fb27165ff921c5a92 (DEFAULT_TABLE table=default.Table_246987d15c034ab5b4be46b8ec5426be [id=7328cd0e6e0e43ecae22597ce9b98a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.564754 749 tablet_service.cc:1468] Processing CreateTablet for tablet 35d999aeadae450184a384ab906c4af8 (DEFAULT_TABLE table=default.Table_f216b85d57b04573a5ec1eb86e843273 [id=a1b9f65573574372a4cd857e3d83b059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.564994 755 tablet_service.cc:1468] Processing CreateTablet for tablet eee8d44dc4be462f9c15c8360f53143a (DEFAULT_TABLE table=default.Table_eefef9f944a5441b8fdc130d0ab579a4 [id=e007600f11084731a0f17613886e3f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.565217 740 tablet_service.cc:1468] Processing CreateTablet for tablet 36a7abd6b4cc4516baf8df29364231e7 (DEFAULT_TABLE table=default.Table_e3e8a33e42a84ab8a6909aa7048291f6 [id=90c1df0aca2a4b7dbc0013406aaa0e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.567881 753 tablet_service.cc:1468] Processing CreateTablet for tablet 697e056a55764ed9846cb02442579d03 (DEFAULT_TABLE table=default.Table_fafd19402c4f4949ba4b7a4bf44473aa [id=ce9ba5c605f446ae86def52b3c4fa506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.573369 757 tablet_service.cc:1468] Processing CreateTablet for tablet 2de995f275184ec9a2ac43aa496d9cad (DEFAULT_TABLE table=default.Table_af4707cebd704b4ba5a48ab600ab26f0 [id=e64bbc424a344192a4b02b9eb2d8695f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.574590 750 tablet_service.cc:1468] Processing CreateTablet for tablet 217545a8120a42808dfa71650ed829e0 (DEFAULT_TABLE table=default.Table_6f92518779264d63a2ed0d8468186644 [id=de6e0ab6513d4f82bab83b4234553393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.575583 746 tablet_service.cc:1468] Processing CreateTablet for tablet a2df4e3918614ff0a6644d56c666d764 (DEFAULT_TABLE table=default.Table_57567ad8836042d783d3e6be3438923e [id=80320f4558b644c0b49889bcf23c3e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.576421 751 tablet_service.cc:1468] Processing CreateTablet for tablet 24e9caef62f64a4281789aba8f83386a (DEFAULT_TABLE table=default.Table_20c6268f43c64a5480574ad87d428a7c [id=1b897d7a629e4a26998e86e4f1ebb33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.596908 1156 ts_tablet_manager.cc:1905] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.578020 756 tablet_service.cc:1468] Processing CreateTablet for tablet 198d774683a34021be94b8057a5ae5ff (DEFAULT_TABLE table=default.Table_6aad014bd6104b998f834208ad803fdb [id=98ca76427dba4c60acd108dc144ca668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.580111 743 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8 (DEFAULT_TABLE table=default.Table_44f5de58c2f14c7b8c1981d2f735bccd [id=f18fa87c934c430d900e532318700a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.581317 742 tablet_service.cc:1468] Processing CreateTablet for tablet 15aef4e26b474f37a4a42e7a7cdf68a1 (DEFAULT_TABLE table=default.Table_aa685973eace4bf5ab1a4d4cbaef51c1 [id=8459cc3f8a5f40f6999b9782d78ffcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.582686 748 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5d7cc320ca455e87d16a78c5f0407d (DEFAULT_TABLE table=default.Table_4622cfe68f7a49a8a6d84d12a686dea7 [id=84672e4ccf754d5d936063ca58a7b807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.584689 747 tablet_service.cc:1468] Processing CreateTablet for tablet 9bbff29d38bf437da3c8273ba1bbec3f (DEFAULT_TABLE table=default.Table_c0c3a2b64e794d63a3c9bce071cf5b97 [id=8a885d9b704b427f9426513b090967dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.585948 744 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec472cd71f04f21aa47b1dfbb32288d (DEFAULT_TABLE table=default.Table_aa8289c495a049b59d261595709a2994 [id=2f803c6780364ab285fa7db60dd77095]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.586606 752 tablet_service.cc:1468] Processing CreateTablet for tablet 332a2808e00c4c52afa3931cb8c53516 (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.598320 2454 leader_election.cc:290] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:58.598866 880 tablet_service.cc:1468] Processing CreateTablet for tablet 697e056a55764ed9846cb02442579d03 (DEFAULT_TABLE table=default.Table_fafd19402c4f4949ba4b7a4bf44473aa [id=ce9ba5c605f446ae86def52b3c4fa506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.512763 2371 tablet.cc:1722] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:58.599238 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 697e056a55764ed9846cb02442579d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.599488 893 tablet_service.cc:1468] Processing CreateTablet for tablet 2de995f275184ec9a2ac43aa496d9cad (DEFAULT_TABLE table=default.Table_af4707cebd704b4ba5a48ab600ab26f0 [id=e64bbc424a344192a4b02b9eb2d8695f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.599645 896 tablet_service.cc:1468] Processing CreateTablet for tablet 36a7abd6b4cc4516baf8df29364231e7 (DEFAULT_TABLE table=default.Table_e3e8a33e42a84ab8a6909aa7048291f6 [id=90c1df0aca2a4b7dbc0013406aaa0e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.599766 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de995f275184ec9a2ac43aa496d9cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.599972 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a7abd6b4cc4516baf8df29364231e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.600418 881 tablet_service.cc:1468] Processing CreateTablet for tablet eee8d44dc4be462f9c15c8360f53143a (DEFAULT_TABLE table=default.Table_eefef9f944a5441b8fdc130d0ab579a4 [id=e007600f11084731a0f17613886e3f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.600886 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee8d44dc4be462f9c15c8360f53143a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.602128 890 tablet_service.cc:1468] Processing CreateTablet for tablet 707ec397649f475fb27165ff921c5a92 (DEFAULT_TABLE table=default.Table_246987d15c034ab5b4be46b8ec5426be [id=7328cd0e6e0e43ecae22597ce9b98a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.602370 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707ec397649f475fb27165ff921c5a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.606742 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 198d774683a34021be94b8057a5ae5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.608665 1156 ts_tablet_manager.cc:1918] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.608739 1156 log.cc:1199] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/2249d5cba1544653b088312e7d74802e
I20250629 01:51:58.608966 1156 ts_tablet_manager.cc:1939] T 2249d5cba1544653b088312e7d74802e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.609581 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1191) took 1269 ms. Trace:
I20250629 01:51:58.609648 1156 rpcz_store.cc:276] 0629 01:51:57.340509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.359364 (+ 18855us) service_pool.cc:225] Handling call
0629 01:51:58.501893 (+1142529us) ts_tablet_manager.cc:1145] Deleting tablet 2249d5cba1544653b088312e7d74802e
0629 01:51:58.501902 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.606100 (+104198us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.608604 (+ 2504us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.609565 (+ 961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1142392}
I20250629 01:51:58.610069 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1193) took 1093 ms. Trace:
I20250629 01:51:58.610147 1156 rpcz_store.cc:276] 0629 01:51:57.516457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.516510 (+ 53us) service_pool.cc:225] Handling call
0629 01:51:58.609970 (+1093460us) ts_tablet_manager.cc:1145] Deleting tablet 49844d26db6447de8c323376c7756437
0629 01:51:58.609978 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.610054 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1093278}
I20250629 01:51:58.610513 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1194) took 1081 ms. Trace:
I20250629 01:51:58.610572 1156 rpcz_store.cc:276] 0629 01:51:57.528878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.529184 (+ 306us) service_pool.cc:225] Handling call
0629 01:51:58.610430 (+1081246us) ts_tablet_manager.cc:1145] Deleting tablet d387af53209147a6bac7afeda9a411ab
0629 01:51:58.610438 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.610500 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081072}
I20250629 01:51:58.610797 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec472cd71f04f21aa47b1dfbb32288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.610920 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1195) took 1080 ms. Trace:
I20250629 01:51:58.610981 1156 rpcz_store.cc:276] 0629 01:51:57.530126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.530485 (+ 359us) service_pool.cc:225] Handling call
0629 01:51:58.610840 (+1080355us) ts_tablet_manager.cc:1145] Deleting tablet 9c90ec6864664fe48d8b9a92837b748c
0629 01:51:58.610848 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.610908 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080188}
I20250629 01:51:58.611279 1156 tablet_replica.cc:331] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.611426 1156 raft_consensus.cc:2241] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.611536 1156 raft_consensus.cc:2270] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.611920 1156 ts_tablet_manager.cc:1905] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.612583 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332a2808e00c4c52afa3931cb8c53516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.614912 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5d7cc320ca455e87d16a78c5f0407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.616824 1156 ts_tablet_manager.cc:1918] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.616919 1156 log.cc:1199] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:51:58.617147 1156 ts_tablet_manager.cc:1939] T cb7314fbdfc74b1c8d9d1be9c05244cb P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.617971 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.619091 2458 raft_consensus.cc:491] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.619202 2458 raft_consensus.cc:513] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.619693 2458 leader_election.cc:290] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.620309 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15aef4e26b474f37a4a42e7a7cdf68a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.623492 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 697e056a55764ed9846cb02442579d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.624276 1155 tablet_bootstrap.cc:492] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.626118 1155 tablet_bootstrap.cc:654] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.626731 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a7abd6b4cc4516baf8df29364231e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.630004 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee8d44dc4be462f9c15c8360f53143a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.632658 1155 tablet_bootstrap.cc:492] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.632752 1155 ts_tablet_manager.cc:1397] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20250629 01:51:58.633134 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d999aeadae450184a384ab906c4af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.633407 1155 raft_consensus.cc:357] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.633538 1155 raft_consensus.cc:383] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.633579 1155 raft_consensus.cc:738] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.633708 1155 consensus_queue.cc:260] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.635907 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2df4e3918614ff0a6644d56c666d764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.636305 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de995f275184ec9a2ac43aa496d9cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.637642 1155 ts_tablet_manager.cc:1428] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:58.637879 1155 tablet_bootstrap.cc:492] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.639591 1155 tablet_bootstrap.cc:654] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.639894 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217545a8120a42808dfa71650ed829e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.640039 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707ec397649f475fb27165ff921c5a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.640223 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e9caef62f64a4281789aba8f83386a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.642181 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbff29d38bf437da3c8273ba1bbec3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.644948 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.646853 1155 tablet_bootstrap.cc:492] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.646941 1155 ts_tablet_manager.cc:1397] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250629 01:51:58.647485 1155 raft_consensus.cc:357] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:58.652391 1155 raft_consensus.cc:383] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:51:58.653553 986 leader_election.cc:343] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:58.666260 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 49844d26db6447de8c323376c7756437 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49844d26db6447de8c323376c7756437 already in progress: creating tablet
W20250629 01:51:58.666466 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet d387af53209147a6bac7afeda9a411ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d387af53209147a6bac7afeda9a411ab already in progress: creating tablet
W20250629 01:51:58.666663 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9c90ec6864664fe48d8b9a92837b748c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c90ec6864664fe48d8b9a92837b748c already in progress: creating tablet
I20250629 01:51:58.669445 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1196) took 1123 ms. Trace:
I20250629 01:51:58.669545 1156 rpcz_store.cc:276] 0629 01:51:57.545771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.545822 (+ 51us) service_pool.cc:225] Handling call
0629 01:51:58.611236 (+1065414us) ts_tablet_manager.cc:1145] Deleting tablet cb7314fbdfc74b1c8d9d1be9c05244cb
0629 01:51:58.611243 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.614360 (+ 3117us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.616766 (+ 2406us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.669425 (+ 52659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1065279}
I20250629 01:51:58.669860 1156 tablet_replica.cc:331] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.669988 1156 raft_consensus.cc:2241] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.670083 1156 raft_consensus.cc:2270] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.670444 1156 ts_tablet_manager.cc:1905] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.675331 1156 ts_tablet_manager.cc:1918] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.675457 1156 log.cc:1199] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:51:58.675874 1156 ts_tablet_manager.cc:1939] T 8a880c9bb4b94d3b9b12f28d48565f06 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.675936 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet cb7314fbdfc74b1c8d9d1be9c05244cb (table default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]) successfully deleted
I20250629 01:51:58.676647 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1197) took 1130 ms. Trace:
I20250629 01:51:58.676792 1156 rpcz_store.cc:276] 0629 01:51:57.545897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.545937 (+ 40us) service_pool.cc:225] Handling call
0629 01:51:58.669816 (+1123879us) ts_tablet_manager.cc:1145] Deleting tablet 8a880c9bb4b94d3b9b12f28d48565f06
0629 01:51:58.669826 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.673013 (+ 3187us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.675274 (+ 2261us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.676629 (+ 1355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123746}
I20250629 01:51:58.677347 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8a880c9bb4b94d3b9b12f28d48565f06 (table default.Table_0ed9b0a279de49078207546c6ddbc21f [id=4457fae28052479e81efac2f717c1bb6]) successfully deleted
I20250629 01:51:58.677697 1156 tablet_replica.cc:331] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.677935 1156 raft_consensus.cc:2241] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.678114 1156 raft_consensus.cc:2270] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.679198 1156 ts_tablet_manager.cc:1905] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.684476 2458 raft_consensus.cc:491] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.684588 2458 raft_consensus.cc:513] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.685077 2458 leader_election.cc:290] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.685652 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3bdc8bda8a940be865679a05b90bd08" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:58.686059 986 leader_election.cc:343] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.693388 913 raft_consensus.cc:3058] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.697031 1155 raft_consensus.cc:738] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.698031 1155 consensus_queue.cc:260] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:58.698812 1155 ts_tablet_manager.cc:1428] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.052s user 0.016s sys 0.000s
I20250629 01:51:58.699318 1155 tablet_bootstrap.cc:492] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.699859 1156 ts_tablet_manager.cc:1918] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.700560 1156 log.cc:1199] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/20586ba2af9e4cb7adec5ca6c3b76e42
I20250629 01:51:58.701110 1156 ts_tablet_manager.cc:1939] T 20586ba2af9e4cb7adec5ca6c3b76e42 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.701188 886 tablet_service.cc:1468] Processing CreateTablet for tablet 217545a8120a42808dfa71650ed829e0 (DEFAULT_TABLE table=default.Table_6f92518779264d63a2ed0d8468186644 [id=de6e0ab6513d4f82bab83b4234553393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.702459 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1198) took 1156 ms. Trace:
I20250629 01:51:58.702553 1156 rpcz_store.cc:276] 0629 01:51:57.546278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.547070 (+ 792us) service_pool.cc:225] Handling call
0629 01:51:58.677650 (+1130580us) ts_tablet_manager.cc:1145] Deleting tablet 20586ba2af9e4cb7adec5ca6c3b76e42
0629 01:51:58.677661 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.689763 (+ 12102us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.699802 (+ 10039us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.702442 (+ 2640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1130435}
I20250629 01:51:58.702922 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1200) took 1084 ms. Trace:
I20250629 01:51:58.702996 1156 rpcz_store.cc:276] 0629 01:51:57.618882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.618945 (+ 63us) service_pool.cc:225] Handling call
0629 01:51:58.702812 (+1083867us) ts_tablet_manager.cc:1145] Deleting tablet 027d08ece22d4b97a3c31e68bcaa396a
0629 01:51:58.702823 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.702909 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083676}
I20250629 01:51:58.703225 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 20586ba2af9e4cb7adec5ca6c3b76e42 (table default.Table_380dba3955bd458caa9a65e48c00f407 [id=77b6e244496149c5a3e46fdfb5bbe791]) successfully deleted
W20250629 01:51:58.706284 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 027d08ece22d4b97a3c31e68bcaa396a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 027d08ece22d4b97a3c31e68bcaa396a already in progress: creating tablet
I20250629 01:51:58.706918 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1201) took 1053 ms. Trace:
I20250629 01:51:58.707237 1156 rpcz_store.cc:276] 0629 01:51:57.653094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.653143 (+ 49us) service_pool.cc:225] Handling call
0629 01:51:58.706364 (+1053221us) ts_tablet_manager.cc:1145] Deleting tablet 2660c211d65f41b0a50b5a6f7d8be48a
0629 01:51:58.706374 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.706899 (+ 525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053045}
I20250629 01:51:58.708194 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1202) took 1020 ms. Trace:
W20250629 01:51:58.708834 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2660c211d65f41b0a50b5a6f7d8be48a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2660c211d65f41b0a50b5a6f7d8be48a already in progress: creating tablet
I20250629 01:51:58.708884 1156 rpcz_store.cc:276] 0629 01:51:57.688209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.688264 (+ 55us) service_pool.cc:225] Handling call
0629 01:51:58.708073 (+1019809us) ts_tablet_manager.cc:1145] Deleting tablet 8d85b9a0e5854e9e9fc16b98143c1cc0
0629 01:51:58.708082 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.708178 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1019593}
W20250629 01:51:58.709661 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 already in progress: creating tablet
I20250629 01:51:58.709725 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1203) took 1021 ms. Trace:
I20250629 01:51:58.709798 1156 rpcz_store.cc:276] 0629 01:51:57.688463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.689343 (+ 880us) service_pool.cc:225] Handling call
0629 01:51:58.709621 (+1020278us) ts_tablet_manager.cc:1145] Deleting tablet 4de6fca85fb643b998be42e9bc7bdbb6
0629 01:51:58.709631 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.709710 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020086}
W20250629 01:51:58.710986 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4de6fca85fb643b998be42e9bc7bdbb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4de6fca85fb643b998be42e9bc7bdbb6 already in progress: creating tablet
I20250629 01:51:58.711088 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1204) took 1021 ms. Trace:
I20250629 01:51:58.711141 1156 rpcz_store.cc:276] 0629 01:51:57.690051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.690090 (+ 39us) service_pool.cc:225] Handling call
0629 01:51:58.710988 (+1020898us) ts_tablet_manager.cc:1145] Deleting tablet b6d60a3c1bc642969ced8c712e5af1c1
0629 01:51:58.710998 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.711073 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1020729}
W20250629 01:51:58.712208 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b6d60a3c1bc642969ced8c712e5af1c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d60a3c1bc642969ced8c712e5af1c1 already in progress: creating tablet
W20250629 01:51:58.712456 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 28da32ab17024f519105c2490efd9136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28da32ab17024f519105c2490efd9136 already in progress: creating tablet
I20250629 01:51:58.712915 1156 tablet_replica.cc:331] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:58.713239 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 58e5cb8c50734d9fbb2b09e764deb11b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58e5cb8c50734d9fbb2b09e764deb11b already in progress: creating tablet
I20250629 01:51:58.713522 1156 raft_consensus.cc:2241] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.714010 1156 raft_consensus.cc:2270] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.714695 1156 ts_tablet_manager.cc:1905] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.700676 1155 tablet_bootstrap.cc:654] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.719098 1155 tablet_bootstrap.cc:492] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.719247 1155 ts_tablet_manager.cc:1397] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250629 01:51:58.719911 1155 raft_consensus.cc:357] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.720139 1155 raft_consensus.cc:383] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.720214 1155 raft_consensus.cc:738] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.720391 1155 consensus_queue.cc:260] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.720803 1155 ts_tablet_manager.cc:1428] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:58.721047 1155 tablet_bootstrap.cc:492] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.722522 1155 tablet_bootstrap.cc:654] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.724359 891 tablet_service.cc:1468] Processing CreateTablet for tablet 24e9caef62f64a4281789aba8f83386a (DEFAULT_TABLE table=default.Table_20c6268f43c64a5480574ad87d428a7c [id=1b897d7a629e4a26998e86e4f1ebb33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.726135 885 tablet_service.cc:1468] Processing CreateTablet for tablet 198d774683a34021be94b8057a5ae5ff (DEFAULT_TABLE table=default.Table_6aad014bd6104b998f834208ad803fdb [id=98ca76427dba4c60acd108dc144ca668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.726611 888 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8 (DEFAULT_TABLE table=default.Table_44f5de58c2f14c7b8c1981d2f735bccd [id=f18fa87c934c430d900e532318700a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.726918 883 tablet_service.cc:1468] Processing CreateTablet for tablet 15aef4e26b474f37a4a42e7a7cdf68a1 (DEFAULT_TABLE table=default.Table_aa685973eace4bf5ab1a4d4cbaef51c1 [id=8459cc3f8a5f40f6999b9782d78ffcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.728217 892 tablet_service.cc:1468] Processing CreateTablet for tablet a2df4e3918614ff0a6644d56c666d764 (DEFAULT_TABLE table=default.Table_57567ad8836042d783d3e6be3438923e [id=80320f4558b644c0b49889bcf23c3e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.729902 884 tablet_service.cc:1468] Processing CreateTablet for tablet 332a2808e00c4c52afa3931cb8c53516 (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.729993 895 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5d7cc320ca455e87d16a78c5f0407d (DEFAULT_TABLE table=default.Table_4622cfe68f7a49a8a6d84d12a686dea7 [id=84672e4ccf754d5d936063ca58a7b807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.730487 878 tablet_service.cc:1468] Processing CreateTablet for tablet 9bbff29d38bf437da3c8273ba1bbec3f (DEFAULT_TABLE table=default.Table_c0c3a2b64e794d63a3c9bce071cf5b97 [id=8a885d9b704b427f9426513b090967dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.730928 879 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec472cd71f04f21aa47b1dfbb32288d (DEFAULT_TABLE table=default.Table_aa8289c495a049b59d261595709a2994 [id=2f803c6780364ab285fa7db60dd77095]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.731475 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.731793 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5e8b8e4d42f43c78f34102b76b4b11a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:58.733409 850 leader_election.cc:304] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:51:58.734020 988 leader_election.cc:343] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.734149 988 leader_election.cc:304] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.734395 2458 raft_consensus.cc:2747] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.734548 2454 raft_consensus.cc:2747] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.733781 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3bdc8bda8a940be865679a05b90bd08" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.738454 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217545a8120a42808dfa71650ed829e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.738629 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec472cd71f04f21aa47b1dfbb32288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.738790 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 198d774683a34021be94b8057a5ae5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.739471 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2df4e3918614ff0a6644d56c666d764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.739624 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5d7cc320ca455e87d16a78c5f0407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.740069 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332a2808e00c4c52afa3931cb8c53516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.740587 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbff29d38bf437da3c8273ba1bbec3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.743299 1156 ts_tablet_manager.cc:1918] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:58.744822 988 leader_election.cc:343] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.744926 1156 log.cc:1199] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9cfff25bea764e62954e49611b71703c
I20250629 01:51:58.744956 988 leader_election.cc:304] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.745152 1156 ts_tablet_manager.cc:1939] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.745361 2458 raft_consensus.cc:2747] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.746846 1156 tablet_replica.cc:331] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.747287 1156 raft_consensus.cc:2241] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.747761 1156 raft_consensus.cc:2270] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.748505 912 raft_consensus.cc:3058] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.748895 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.749528 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e9caef62f64a4281789aba8f83386a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.749845 1156 ts_tablet_manager.cc:1905] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.750138 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15aef4e26b474f37a4a42e7a7cdf68a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:58.761874 1155 tablet_bootstrap.cc:492] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.763296 1155 ts_tablet_manager.cc:1397] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.001s
I20250629 01:51:58.764341 1155 raft_consensus.cc:357] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.765769 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9cfff25bea764e62954e49611b71703c (table default.Table_bf7b726d4c38474994e10c22d1aa6076 [id=875222c495154458ae2052f4ca5d220f]) successfully deleted
I20250629 01:51:58.766187 1155 raft_consensus.cc:383] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.766247 1155 raft_consensus.cc:738] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.766404 1155 consensus_queue.cc:260] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.766767 1155 ts_tablet_manager.cc:1428] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:51:58.766961 1155 tablet_bootstrap.cc:492] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.768221 1155 tablet_bootstrap.cc:654] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.772539 1156 ts_tablet_manager.cc:1918] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.775375 1156 log.cc:1199] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f4b7d0a44bbe4d8c81e8d0c444dbb55b
I20250629 01:51:58.775796 1156 ts_tablet_manager.cc:1939] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.778198 1156 tablet_replica.cc:331] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.778391 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b (table default.Table_2acf6246904d41ab94aad9dec7df2291 [id=df90a81df72e4bc09675d9b0203888fc]) successfully deleted
W20250629 01:51:58.778723 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f2336702c5e941ce994df6edb1970192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2336702c5e941ce994df6edb1970192 already in progress: creating tablet
W20250629 01:51:58.779040 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 45461dc364d847d0b0272ac714559008 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45461dc364d847d0b0272ac714559008 already in progress: creating tablet
I20250629 01:51:58.778983 1155 tablet_bootstrap.cc:492] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.779353 1155 ts_tablet_manager.cc:1397] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20250629 01:51:58.779363 1156 raft_consensus.cc:2241] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.779881 1156 raft_consensus.cc:2270] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.780176 1155 raft_consensus.cc:357] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.780334 1155 raft_consensus.cc:383] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.780390 1155 raft_consensus.cc:738] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.780575 1155 consensus_queue.cc:260] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.781025 1155 ts_tablet_manager.cc:1428] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:58.781348 1155 tablet_bootstrap.cc:492] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.781085 1156 ts_tablet_manager.cc:1905] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.782030 2049 raft_consensus.cc:491] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.782209 2049 raft_consensus.cc:513] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.782748 1155 tablet_bootstrap.cc:654] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.782721 2049 leader_election.cc:290] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.794394 1156 ts_tablet_manager.cc:1918] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.783851 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "daf312e60c014fe698d3ec9f1a1b769c" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:58.815315 1156 log.cc:1199] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/74b3badd684d45fa88dc8cbcfc61001a
W20250629 01:51:58.803164 712 leader_election.cc:343] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.815819 1156 ts_tablet_manager.cc:1939] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.817327 1155 tablet_bootstrap.cc:492] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:58.817432 1155 ts_tablet_manager.cc:1397] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.009s
I20250629 01:51:58.818166 1155 raft_consensus.cc:357] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.818356 1155 raft_consensus.cc:383] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.818420 1155 raft_consensus.cc:738] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.818617 1155 consensus_queue.cc:260] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.822782 1155 ts_tablet_manager.cc:1428] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:51:58.872836 2487 tablet.cc:1722] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:58.900295 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1216) took 1001 ms. Trace:
I20250629 01:51:58.900401 1156 rpcz_store.cc:276] 0629 01:51:57.898455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.898513 (+ 58us) service_pool.cc:225] Handling call
0629 01:51:58.778154 (+879641us) ts_tablet_manager.cc:1145] Deleting tablet 74b3badd684d45fa88dc8cbcfc61001a
0629 01:51:58.778162 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.791393 (+ 13231us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.794337 (+ 2944us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.900265 (+105928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":879468}
I20250629 01:51:58.901083 1156 tablet_replica.cc:331] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:58.901291 1156 raft_consensus.cc:2241] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:58.901423 1156 raft_consensus.cc:2270] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:58.901934 1156 ts_tablet_manager.cc:1905] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:58.900750 1155 tablet_bootstrap.cc:492] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:58.905423 1155 tablet_bootstrap.cc:654] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:58.911403 1156 ts_tablet_manager.cc:1918] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.911691 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 74b3badd684d45fa88dc8cbcfc61001a (table default.Table_75f56cc1109c41f2a0a3ba83902b88ea [id=fcdd225098994f2f8c425a8fc3165d24]) successfully deleted
I20250629 01:51:58.914302 1156 log.cc:1199] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/68cedbd093b4420d8f7b29219ce8f77b
I20250629 01:51:58.914796 1156 ts_tablet_manager.cc:1939] T 68cedbd093b4420d8f7b29219ce8f77b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:58.915458 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1217) took 1016 ms. Trace:
I20250629 01:51:58.915877 912 raft_consensus.cc:3058] T 38f831a6dd68493382d5f2920d0794a8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:58.915933 1156 rpcz_store.cc:276] 0629 01:51:57.898813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.898880 (+ 67us) service_pool.cc:225] Handling call
0629 01:51:58.901037 (+1002157us) ts_tablet_manager.cc:1145] Deleting tablet 68cedbd093b4420d8f7b29219ce8f77b
0629 01:51:58.901047 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.908129 (+ 7082us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.911322 (+ 3193us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.915442 (+ 4120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1002016}
I20250629 01:51:58.916776 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 68cedbd093b4420d8f7b29219ce8f77b (table default.Table_31aef3786c3841c096a18ca7abfa40df [id=b46cf5dd7d524d0cb23f236b318eafca]) successfully deleted
W20250629 01:51:58.917608 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ec0eeb0ef02c416abc5665f058c6bd7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0eeb0ef02c416abc5665f058c6bd7f already in progress: creating tablet
W20250629 01:51:58.918758 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2a876b3db3904a4297b5b13a9325b1c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a876b3db3904a4297b5b13a9325b1c3 already in progress: creating tablet
I20250629 01:51:58.919939 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a45ac1d6a41941178905d51452d4c4fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:58.920542 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a45ac1d6a41941178905d51452d4c4fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:58.921653 2436 raft_consensus.cc:491] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.921774 2436 raft_consensus.cc:513] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.921808 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4" } modify_external_catalogs: true
I20250629 01:51:58.921949 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3d30bb7ae9fa4749a0bed1f62056c6c4" } modify_external_catalogs: true
I20250629 01:51:58.922325 2436 leader_election.cc:290] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.923271 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40592f11f430469f9c8e82eba25b38ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.924430 712 leader_election.cc:343] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.933398 447 catalog_manager.cc:5582] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:58.933737 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:51:58.933959 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:51:58.934212 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:58.956202 1159 tablet_bootstrap.cc:492] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:58.956310 1159 ts_tablet_manager.cc:1397] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.475s user 0.045s sys 0.097s
I20250629 01:51:58.957010 1159 raft_consensus.cc:357] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.957175 1159 raft_consensus.cc:383] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.957229 1159 raft_consensus.cc:738] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.957440 1159 consensus_queue.cc:260] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.957990 1159 ts_tablet_manager.cc:1428] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:58.958257 1159 tablet_bootstrap.cc:492] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:58.962213 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2fcead51fd9b43d0a52c4669a5e23fe5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.964383 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40592f11f430469f9c8e82eba25b38ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:58.964524 1047 raft_consensus.cc:2466] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:58.966701 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 06e31520d3ad4c3395a59049755a4725 (table default.Table_2e69d514f50649f79824a1cb1da60444 [id=d843c94cf5f847249d82ea7944980d89]) successfully deleted
W20250629 01:51:58.966971 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 58e5cb8c50734d9fbb2b09e764deb11b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58e5cb8c50734d9fbb2b09e764deb11b already in progress: creating tablet
W20250629 01:51:58.967175 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d85b9a0e5854e9e9fc16b98143c1cc0 already in progress: creating tablet
W20250629 01:51:58.967353 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4de6fca85fb643b998be42e9bc7bdbb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4de6fca85fb643b998be42e9bc7bdbb6 already in progress: creating tablet
W20250629 01:51:58.967541 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b6d60a3c1bc642969ced8c712e5af1c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d60a3c1bc642969ced8c712e5af1c1 already in progress: creating tablet
W20250629 01:51:58.977376 1154 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.037s sys 0.131s
I20250629 01:51:58.977385 2436 raft_consensus.cc:491] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.977501 2436 raft_consensus.cc:513] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.977952 2436 leader_election.cc:290] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:58.978075 1154 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.593s user 0.037s sys 0.131s
I20250629 01:51:58.979156 1154 tablet_bootstrap.cc:492] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:58.979611 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d9ef77181ba48f3a6de16b945c8df1a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.980042 712 leader_election.cc:343] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:58.980237 2454 raft_consensus.cc:491] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.980331 2454 raft_consensus.cc:513] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.980820 2454 leader_election.cc:290] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:58.981770 850 leader_election.cc:343] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e3b17172ab8b4e0dad36ebaac776dc03
W20250629 01:51:58.981889 850 leader_election.cc:343] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e3b17172ab8b4e0dad36ebaac776dc03
I20250629 01:51:58.981937 850 leader_election.cc:304] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:58.982192 1154 ts_tablet_manager.cc:1397] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.599s user 0.038s sys 0.132s
I20250629 01:51:58.982437 2454 raft_consensus.cc:2747] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:58.983044 1154 raft_consensus.cc:357] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.984174 2487 tablet.cc:1722] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:58.987751 2436 raft_consensus.cc:491] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:58.987881 2436 raft_consensus.cc:513] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.988333 2436 leader_election.cc:290] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:58.989722 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6092e19acf974f2dad5a903da72944ba" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:58.990116 712 leader_election.cc:343] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:58.990180 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 28da32ab17024f519105c2490efd9136 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28da32ab17024f519105c2490efd9136 already in progress: creating tablet
I20250629 01:51:58.991977 1154 raft_consensus.cc:383] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:58.992105 1154 raft_consensus.cc:738] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:58.992340 1154 consensus_queue.cc:260] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:58.979352 1158 ts_tablet_manager.cc:1918] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:58.992621 1158 log.cc:1199] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e3b94f8250a143b096703f2b0386e3cd
I20250629 01:51:58.992954 1158 ts_tablet_manager.cc:1939] T e3b94f8250a143b096703f2b0386e3cd P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:58.996943 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.61:35873
I20250629 01:51:58.999544 896 tablet_service.cc:1468] Processing CreateTablet for tablet df692a03604444f18cf7b0b337ea3fa5 (DEFAULT_TABLE table=default.Table_1382661b867943bc8431964543552b18 [id=8cf88e839b2c4a969090f8278c34a31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:58.999964 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df692a03604444f18cf7b0b337ea3fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.000497 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1175) took 1213 ms. Trace:
I20250629 01:51:59.004309 890 tablet_service.cc:1468] Processing CreateTablet for tablet 43b33ec0ed684ae99e49878dc7acbb33 (DEFAULT_TABLE table=default.Table_40a12cf7daec42ef90b22c45964e56f6 [id=527140d11ad841888dbe9466e8e9c2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.004691 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b33ec0ed684ae99e49878dc7acbb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.009251 874 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56382 (request call id 167) took 1122 ms. Trace:
I20250629 01:51:59.009394 874 rpcz_store.cc:276] 0629 01:51:57.886368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.886420 (+ 52us) service_pool.cc:225] Handling call
0629 01:51:57.934751 (+ 48331us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x618000163c8c. stack: 00007f9a19ba0316 00007f9a19ba00d0 00007f9a18b963ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9a26b218be 00007f9a2691d071 00007f9a293633b8 00007f9a23aa2a36 00007f9a23aa2765 00007f9a20943c7e 00007f9a20942b5e 00007f9a20948d95 00007f9a209496bf 00007f9a209494e8
0629 01:51:58.604366 (+669615us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x6160000b8354. stack: 00007f9a19ba0316 00007f9a19ba00d0 00007f9a18b963ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9a255d0420 00007f9a293635af 00007f9a23aa2a36 00007f9a23aa2765 00007f9a20943c7e 00007f9a20942b5e 00007f9a20948d95 00007f9a209496bf 00007f9a209494e8 0000000000a7573c
0629 01:51:59.009216 (+404850us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1300464640}
I20250629 01:51:59.014657 1158 rpcz_store.cc:276] 0629 01:51:57.789626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.791655 (+ 2029us) service_pool.cc:225] Handling call
0629 01:51:58.425382 (+633727us) ts_tablet_manager.cc:1145] Deleting tablet e3b94f8250a143b096703f2b0386e3cd
0629 01:51:58.425388 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:58.531547 (+106159us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:58.979261 (+447714us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.000477 (+ 21216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6628,"tablet-delete.queue_time_us":633611}
I20250629 01:51:59.002825 1154 ts_tablet_manager.cc:1428] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20250629 01:51:59.020648 750 tablet_service.cc:1468] Processing CreateTablet for tablet 43b33ec0ed684ae99e49878dc7acbb33 (DEFAULT_TABLE table=default.Table_40a12cf7daec42ef90b22c45964e56f6 [id=527140d11ad841888dbe9466e8e9c2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.021046 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b33ec0ed684ae99e49878dc7acbb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.024189 757 tablet_service.cc:1468] Processing CreateTablet for tablet df692a03604444f18cf7b0b337ea3fa5 (DEFAULT_TABLE table=default.Table_1382661b867943bc8431964543552b18 [id=8cf88e839b2c4a969090f8278c34a31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.024554 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df692a03604444f18cf7b0b337ea3fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.029368 872 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56382 (request call id 169) took 1050 ms. Trace:
I20250629 01:51:59.029474 872 rpcz_store.cc:276] 0629 01:51:57.979331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.979387 (+ 56us) service_pool.cc:225] Handling call
0629 01:51:58.426805 (+447418us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x6160001a2954. stack: 00007f9a19ba0316 00007f9a19ba00d0 00007f9a18b963ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9a255d0420 00007f9a293635af 00007f9a23aa2a36 00007f9a23aa2765 00007f9a20943c7e 00007f9a20942b5e 00007f9a20948d95 00007f9a209496bf 00007f9a209494e8 0000000000a7573c
0629 01:51:58.605133 (+178328us) spinlock_profiling.cc:243] Waited 74.3 ms on lock 0x6160000b8354. stack: 00007f9a19ba0316 00007f9a19ba00d0 00007f9a18b963ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9a255d0420 00007f9a293635af 00007f9a23aa2a36 00007f9a23aa2765 00007f9a20943c7e 00007f9a20942b5e 00007f9a20948d95 00007f9a209496bf 00007f9a209494e8 0000000000a7573c
0629 01:51:58.731791 (+126658us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x6180001e188c. stack: 00007f9a19ba0316 00007f9a19ba00d0 00007f9a18b963ad 0000000000971004 0000000000970fcd 000000000097099a 00007f9a26b218be 00007f9a2691d071 00007f9a293633b8 00007f9a23aa2a36 00007f9a23aa2765 00007f9a20943c7e 00007f9a20942b5e 00007f9a20948d95 00007f9a209496bf 00007f9a209494e8
0629 01:51:59.029331 (+297540us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1294986240}
I20250629 01:51:59.032473 1159 tablet_bootstrap.cc:654] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.035305 2436 consensus_queue.cc:1035] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:59.036134 2436 consensus_queue.cc:1035] T 95247cc778504f4d85d9013f84c59e69 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:59.036289 710 leader_election.cc:304] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:51:59.036453 710 raft_consensus.cc:2684] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:59.036756 1155 tablet_bootstrap.cc:492] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.036890 1155 ts_tablet_manager.cc:1397] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.136s user 0.014s sys 0.008s
W20250629 01:51:59.037281 710 leader_election.cc:343] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9cfff25bea764e62954e49611b71703c
I20250629 01:51:59.037385 710 leader_election.cc:304] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:59.037537 710 raft_consensus.cc:2684] T 9cfff25bea764e62954e49611b71703c P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:59.037540 1155 raft_consensus.cc:357] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.037720 1155 raft_consensus.cc:383] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.037796 1155 raft_consensus.cc:738] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.037994 1155 consensus_queue.cc:260] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.045030 1158 tablet_replica.cc:331] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:59.045202 1158 raft_consensus.cc:2241] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.045337 1158 raft_consensus.cc:2270] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.045785 1158 ts_tablet_manager.cc:1905] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.048385 1154 tablet_bootstrap.cc:492] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:59.049695 1154 tablet_bootstrap.cc:654] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.053691 2436 raft_consensus.cc:491] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.053815 2436 raft_consensus.cc:513] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.054299 2436 leader_election.cc:290] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:59.054569 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a81e6f9711c4d398ae09f33a75566ae" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:59.055018 712 leader_election.cc:343] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.056860 1155 ts_tablet_manager.cc:1428] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250629 01:51:59.057125 1155 tablet_bootstrap.cc:492] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.058621 1155 tablet_bootstrap.cc:654] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.059681 710 leader_election.cc:304] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.059911 2436 raft_consensus.cc:2802] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.059998 2436 raft_consensus.cc:491] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.060063 2436 raft_consensus.cc:3058] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.060453 2433 consensus_queue.cc:1035] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:51:59.060662 2458 raft_consensus.cc:491] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.060757 2458 raft_consensus.cc:513] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.061194 2458 leader_election.cc:290] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:51:59.061527 710 leader_election.cc:343] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 74b3badd684d45fa88dc8cbcfc61001a
I20250629 01:51:59.061623 710 leader_election.cc:304] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:59.061774 710 raft_consensus.cc:2684] T 74b3badd684d45fa88dc8cbcfc61001a P ef5c5979bb9e475f983af7c3e9e0e019: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:51:59.061834 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a81e6f9711c4d398ae09f33a75566ae" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:59.061992 777 raft_consensus.cc:2466] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:59.062280 2436 raft_consensus.cc:513] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.062335 986 leader_election.cc:304] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:59.062525 2458 raft_consensus.cc:2802] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.062579 2458 raft_consensus.cc:491] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.062645 2458 raft_consensus.cc:3058] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.062768 2436 leader_election.cc:290] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.063174 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a81e6f9711c4d398ae09f33a75566ae" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:59.063350 2049 consensus_queue.cc:1035] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:51:59.063616 988 leader_election.cc:343] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.065045 2458 raft_consensus.cc:513] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.066079 710 leader_election.cc:304] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:59.066309 2049 raft_consensus.cc:2747] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.066514 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.066520 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a81e6f9711c4d398ae09f33a75566ae" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:59.066663 777 raft_consensus.cc:3058] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:51:59.067011 712 leader_election.cc:343] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:59.067538 710 leader_election.cc:343] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 06e31520d3ad4c3395a59049755a4725
I20250629 01:51:59.067646 710 leader_election.cc:304] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:59.069361 777 raft_consensus.cc:2466] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:59.070842 1158 ts_tablet_manager.cc:1918] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.071003 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a81e6f9711c4d398ae09f33a75566ae" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.071641 2049 raft_consensus.cc:2747] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.072618 2454 raft_consensus.cc:491] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.072728 2454 raft_consensus.cc:513] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.073160 2454 leader_election.cc:290] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:59.073839 850 leader_election.cc:343] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 2aca26af31ed4bc994f4d62603edc881
W20250629 01:51:59.074222 710 leader_election.cc:343] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:51:59.074776 710 leader_election.cc:304] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:59.075366 2433 raft_consensus.cc:2747] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.075934 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1a5d7cc320ca455e87d16a78c5f0407d
I20250629 01:51:59.076458 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6d7afa4be82a4ae3974bd0b4786d22db
I20250629 01:51:59.076807 454 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56238 (request call id 1409) took 1090 ms. Trace:
I20250629 01:51:59.077023 710 leader_election.cc:304] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.077702 2433 raft_consensus.cc:2802] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.076911 454 rpcz_store.cc:276] 0629 01:51:57.986027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.261649 (+275622us) service_pool.cc:225] Handling call
0629 01:51:58.270453 (+ 8804us) spinlock_profiling.cc:243] Waited 8.59 ms on lock 0x616000040f54. stack: 00007f945d0ff316 00007f945d0ff0d0 00007f945c0f53ad 0000000000971004 0000000000970fcd 00007f946bea014d 00007f946bea0243 00007f9468b63661 00007f94706e4d8d 00007f947086eda6 00007f94676b6316 00007f94676b6045 00007f9463ea2c7e 00007f9463ea1b5e 00007f9463ea7d95 00007f9463ea86bf
0629 01:51:58.270782 (+ 329us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0629 01:51:58.270899 (+ 117us) catalog_manager.cc:2790] Modifying in-memory table state
0629 01:51:58.270938 (+ 39us) catalog_manager.cc:2799] Locking tablets
0629 01:51:58.558809 (+287871us) catalog_manager.cc:2813] Removing table and tablets from system table
0629 01:51:59.076341 (+517532us) catalog_manager.cc:2831] Removing table from by-name map
0629 01:51:59.076399 (+ 58us) catalog_manager.cc:2847] Committing in-memory state
0629 01:51:59.076436 (+ 37us) catalog_manager.cc:2851] Aborting table tasks
0629 01:51:59.076790 (+ 354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0629 01:51:58.971604 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0629 01:51:58.971678 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0629 01:51:58.971682 (+ 4us) write_op.cc:435] Acquired schema lock
0629 01:51:58.971684 (+ 2us) tablet.cc:660] Decoding operations
0629 01:51:58.971754 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0629 01:51:58.971760 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0629 01:51:58.971764 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0629 01:51:58.971849 (+ 85us) tablet.cc:699] Row locks acquired
0629 01:51:58.971852 (+ 3us) write_op.cc:260] PREPARE: finished
0629 01:51:58.971872 (+ 20us) write_op.cc:270] Start()
0629 01:51:58.971890 (+ 18us) write_op.cc:276] Timestamp: P: 1751161918971818 usec, L: 0
0629 01:51:58.971893 (+ 3us) op_driver.cc:348] REPLICATION: starting
0629 01:51:58.971997 (+ 104us) log.cc:844] Serialized 725 byte log entry
0629 01:51:59.075088 (+103091us) op_driver.cc:464] REPLICATION: finished
0629 01:51:59.075600 (+ 512us) write_op.cc:301] APPLY: starting
0629 01:51:59.075608 (+ 8us) tablet.cc:1365] starting BulkCheckPresence
0629 01:51:59.075637 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0629 01:51:59.075656 (+ 19us) tablet.cc:1370] starting ApplyRowOperation cycle
0629 01:51:59.075708 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0629 01:51:59.075732 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0629 01:51:59.075738 (+ 6us) write_op.cc:312] APPLY: finished
0629 01:51:59.075785 (+ 47us) log.cc:844] Serialized 29 byte log entry
0629 01:51:59.075825 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0629 01:51:59.075838 (+ 13us) write_op.cc:454] Released schema lock
0629 01:51:59.075843 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0629 01:51:59.075898 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0629 01:51:59.075901 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":19755904,"child_traces":[["op",{"apply.queue_time_us":478,"apply.run_cpu_time_us":292,"apply.run_wall_time_us":292,"num_ops":2,"prepare.queue_time_us":412345,"prepare.run_cpu_time_us":446,"prepare.run_wall_time_us":687,"replication_time_us":103181}]]}
I20250629 01:51:59.077780 2433 raft_consensus.cc:491] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.077852 2433 raft_consensus.cc:3058] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.078305 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 84cfa2d73c6b4ca2bf41437b196d10b1
I20250629 01:51:59.078750 451 catalog_manager.cc:6652] ReplaceTablet: tablet 84cfa2d73c6b4ca2bf41437b196d10b1 deleted and replaced by tablet b936c61fbbc841278d13fbc9d9860eed
I20250629 01:51:59.079996 2433 raft_consensus.cc:513] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.080710 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d7afa4be82a4ae3974bd0b4786d22db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.080983 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b936c61fbbc841278d13fbc9d9860eed) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.081496 2433 leader_election.cc:290] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.081871 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40592f11f430469f9c8e82eba25b38ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:59.082304 712 leader_election.cc:343] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.070946 2458 leader_election.cc:290] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.071254 986 leader_election.cc:304] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:51:59.084575 988 leader_election.cc:343] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.085244 2458 raft_consensus.cc:2802] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.085343 2458 raft_consensus.cc:695] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:59.085531 2458 consensus_queue.cc:237] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.089187 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5d7cc320ca455e87d16a78c5f0407d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:59.090546 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f1ad1b7de79b48058437a9dea5d30b37 (table default.Table_23b66e043cc746f68172596b5afc01c9 [id=6faf9bc8cce84174a48928674dc25702], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f1ad1b7de79b48058437a9dea5d30b37 consensus state has no leader
I20250629 01:51:59.091118 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d7afa4be82a4ae3974bd0b4786d22db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.092175 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b936c61fbbc841278d13fbc9d9860eed) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:59.093472 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1a5d7cc320ca455e87d16a78c5f0407d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a5d7cc320ca455e87d16a78c5f0407d already in progress: creating tablet
W20250629 01:51:59.093739 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6d7afa4be82a4ae3974bd0b4786d22db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d7afa4be82a4ae3974bd0b4786d22db already in progress: creating tablet
W20250629 01:51:59.094084 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84cfa2d73c6b4ca2bf41437b196d10b1 already in progress: creating tablet
W20250629 01:51:59.094367 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6d7afa4be82a4ae3974bd0b4786d22db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d7afa4be82a4ae3974bd0b4786d22db already in progress: creating tablet
W20250629 01:51:59.094566 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84cfa2d73c6b4ca2bf41437b196d10b1 already in progress: creating tablet
I20250629 01:51:59.095767 1158 log.cc:1199] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/6092e19acf974f2dad5a903da72944ba
I20250629 01:51:59.095913 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cc65674750c64114b999e0e292a851f9
I20250629 01:51:59.096203 1158 ts_tablet_manager.cc:1939] T 6092e19acf974f2dad5a903da72944ba P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:59.096310 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56238 (request call id 1410) took 1018 ms. Trace:
I20250629 01:51:59.096375 452 rpcz_store.cc:276] 0629 01:51:58.077576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.262203 (+184627us) service_pool.cc:225] Handling call
0629 01:51:58.272505 (+ 10302us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x616000040f54. stack: 00007f945d0ff316 00007f945d0ff0d0 00007f945c0f53ad 0000000000971004 0000000000970fcd 00007f946bea014d 00007f946bea0243 00007f9468b63661 00007f94706e4d8d 00007f947086eda6 00007f94676b6316 00007f94676b6045 00007f9463ea2c7e 00007f9463ea1b5e 00007f9463ea7d95 00007f9463ea86bf
0629 01:51:58.272791 (+ 286us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0629 01:51:58.272912 (+ 121us) catalog_manager.cc:2790] Modifying in-memory table state
0629 01:51:58.272962 (+ 50us) catalog_manager.cc:2799] Locking tablets
0629 01:51:58.560019 (+287057us) catalog_manager.cc:2813] Removing table and tablets from system table
0629 01:51:59.095797 (+535778us) catalog_manager.cc:2831] Removing table from by-name map
0629 01:51:59.095854 (+ 57us) catalog_manager.cc:2847] Committing in-memory state
0629 01:51:59.095887 (+ 33us) catalog_manager.cc:2851] Aborting table tasks
0629 01:51:59.096298 (+ 411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0629 01:51:58.973145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0629 01:51:58.973215 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0629 01:51:58.973220 (+ 5us) write_op.cc:435] Acquired schema lock
0629 01:51:58.973222 (+ 2us) tablet.cc:660] Decoding operations
0629 01:51:58.973289 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0629 01:51:58.973295 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0629 01:51:58.973299 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0629 01:51:58.973378 (+ 79us) tablet.cc:699] Row locks acquired
0629 01:51:58.973380 (+ 2us) write_op.cc:260] PREPARE: finished
0629 01:51:58.973401 (+ 21us) write_op.cc:270] Start()
0629 01:51:58.973420 (+ 19us) write_op.cc:276] Timestamp: P: 1751161918973347 usec, L: 0
0629 01:51:58.973422 (+ 2us) op_driver.cc:348] REPLICATION: starting
0629 01:51:59.040610 (+ 67188us) log.cc:844] Serialized 756 byte log entry
0629 01:51:59.094890 (+ 54280us) op_driver.cc:464] REPLICATION: finished
0629 01:51:59.095099 (+ 209us) write_op.cc:301] APPLY: starting
0629 01:51:59.095109 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0629 01:51:59.095137 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0629 01:51:59.095180 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0629 01:51:59.095227 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0629 01:51:59.095249 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0629 01:51:59.095256 (+ 7us) write_op.cc:312] APPLY: finished
0629 01:51:59.095307 (+ 51us) log.cc:844] Serialized 29 byte log entry
0629 01:51:59.095346 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0629 01:51:59.095358 (+ 12us) write_op.cc:454] Released schema lock
0629 01:51:59.095363 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0629 01:51:59.095424 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0629 01:51:59.095428 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":23294848,"child_traces":[["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":301,"apply.run_wall_time_us":301,"num_ops":2,"prepare.queue_time_us":412673,"prepare.run_cpu_time_us":543,"prepare.run_wall_time_us":68145,"raft.queue_time_us":26,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":171,"replication_time_us":121445}]]}
I20250629 01:51:59.096933 2433 raft_consensus.cc:491] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.097033 2433 raft_consensus.cc:513] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.097491 2433 leader_election.cc:290] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.097805 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3b94f8250a143b096703f2b0386e3cd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:59.098173 712 leader_election.cc:343] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.098927 892 tablet_service.cc:1515] Processing DeleteTablet for tablet cc65674750c64114b999e0e292a851f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.099531 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fbc83e65ade946eba813be6e4ff8c3f0" } modify_external_catalogs: true
I20250629 01:51:59.099680 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fbc83e65ade946eba813be6e4ff8c3f0" } modify_external_catalogs: true
I20250629 01:51:59.100554 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7c434928fb344af2aebbe8eb3f5376a2" } new_table_name: "default.Table_91fc3459bd5846eebdbda0ddbd31d768" modify_external_catalogs: true
I20250629 01:51:59.100996 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e3c425798b1f45acaa203c1a4dcad1c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.101401 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3c425798b1f45acaa203c1a4dcad1c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.101715 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2d2750afeab44fda88c390b72b529bcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.102092 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d2750afeab44fda88c390b72b529bcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.102412 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d1dc5307aa7c47738a408a2ebd00a685"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.102788 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1dc5307aa7c47738a408a2ebd00a685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.103394 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4e8a8a6b9cde425c8326fd3609daf30f" } new_table_name: "default.Table_f705303395b14b0daf27cb67a19275a8" modify_external_catalogs: true
W20250629 01:51:59.103636 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet cc65674750c64114b999e0e292a851f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc65674750c64114b999e0e292a851f9 already in progress: creating tablet
I20250629 01:51:59.104449 2017 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40868 (request call id 0) took 6521 ms. Trace:
I20250629 01:51:59.104933 2507 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0b76160e22c46929936cb9682322fae)
I20250629 01:51:59.105150 2507 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:51:59.107636 2507 leader_election.cc:290] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0b76160e22c46929936cb9682322fae (127.0.104.62:34747), f304077528b34c61857a659e5873631d (127.0.104.60:32789)
I20250629 01:51:59.108573 1155 tablet_bootstrap.cc:492] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.108693 1155 ts_tablet_manager.cc:1397] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.003s
I20250629 01:51:59.108794 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b76160e22c46929936cb9682322fae. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:51:59.109019 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [sys.catalog]: This master's current role is: FOLLOWER
I20250629 01:51:59.110277 1154 tablet_bootstrap.cc:492] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:59.111085 1154 ts_tablet_manager.cc:1397] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.008s
I20250629 01:51:59.111932 1154 raft_consensus.cc:357] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.112507 1155 raft_consensus.cc:357] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.113695 1155 raft_consensus.cc:383] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.113822 1155 raft_consensus.cc:738] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.114145 1155 consensus_queue.cc:260] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.114825 1155 ts_tablet_manager.cc:1428] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250629 01:51:59.114872 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5d7cc320ca455e87d16a78c5f0407d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.115090 1154 raft_consensus.cc:383] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.122810 1159 tablet_bootstrap.cc:492] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.122967 1159 ts_tablet_manager.cc:1397] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.165s user 0.005s sys 0.000s
I20250629 01:51:59.125761 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1e87caeaa536423685a228252bb9a901
I20250629 01:51:59.126420 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fd765b36942a425a89bfdd90ae5efd35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.126901 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd765b36942a425a89bfdd90ae5efd35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.129186 750 tablet_service.cc:1515] Processing DeleteTablet for tablet cc65674750c64114b999e0e292a851f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.129451 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e87caeaa536423685a228252bb9a901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.130064 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e87caeaa536423685a228252bb9a901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.120190 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 707ec397649f475fb27165ff921c5a92 (DEFAULT_TABLE table=default.Table_246987d15c034ab5b4be46b8ec5426be [id=7328cd0e6e0e43ecae22597ce9b98a53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.120215 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 35d999aeadae450184a384ab906c4af8 (DEFAULT_TABLE table=default.Table_f216b85d57b04573a5ec1eb86e843273 [id=a1b9f65573574372a4cd857e3d83b059]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.124374 1030 tablet_service.cc:1468] Processing CreateTablet for tablet eee8d44dc4be462f9c15c8360f53143a (DEFAULT_TABLE table=default.Table_eefef9f944a5441b8fdc130d0ab579a4 [id=e007600f11084731a0f17613886e3f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.130275 1154 raft_consensus.cc:738] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.130870 1154 consensus_queue.cc:260] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.131450 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 36a7abd6b4cc4516baf8df29364231e7 (DEFAULT_TABLE table=default.Table_e3e8a33e42a84ab8a6909aa7048291f6 [id=90c1df0aca2a4b7dbc0013406aaa0e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.134874 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae with delete_type TABLET_DATA_DELETED (replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.136572 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 819ed44d98f14075b9a67b88dc44e86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.137333 1159 raft_consensus.cc:357] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.137490 1159 raft_consensus.cc:383] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.137545 1159 raft_consensus.cc:738] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.137758 1155 tablet_bootstrap.cc:492] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.137735 1159 consensus_queue.cc:260] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:59.138150 850 leader_election.cc:343] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2aca26af31ed4bc994f4d62603edc881
I20250629 01:51:59.138283 850 leader_election.cc:304] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.138608 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.139647 1155 tablet_bootstrap.cc:654] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.131470 1154 ts_tablet_manager.cc:1428] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20250629 01:51:59.142257 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_44f5de58c2f14c7b8c1981d2f735bccd" } new_table_name: "default.Table_3a778aeca879461e80de6fb8e3c8de02" modify_external_catalogs: true
I20250629 01:51:59.142702 2531 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250629 01:51:59.143503 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet e5e8b8e4d42f43c78f34102b76b4b11a (table default.Table_904b8b7a3ab1414a9bd29274648bbb86 [id=e991e29b3d9d4dcc96912754565336f8]). Expected version 1 got 0
I20250629 01:51:59.143635 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:59.143756 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:59.143909 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 176bd3e921c74f9a93d4b2f1c0fc97a7 (table default.Table_4d28e4ec92114a31856c3992b0728729 [id=3d577c959e5b4549a57fa4cdc320a0d6]). Expected version 5 got 0
I20250629 01:51:59.144044 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:59.148993 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:59.149359 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:59.149637 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:59.149775 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:51:59.149895 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:59.150094 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:51:59.150316 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:51:59.150595 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943]). Expected version 5 got 0
W20250629 01:51:59.152212 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet cc65674750c64114b999e0e292a851f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc65674750c64114b999e0e292a851f9 already in progress: creating tablet
I20250629 01:51:59.153980 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 707ec397649f475fb27165ff921c5a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.154238 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 697e056a55764ed9846cb02442579d03 (DEFAULT_TABLE table=default.Table_fafd19402c4f4949ba4b7a4bf44473aa [id=ce9ba5c605f446ae86def52b3c4fa506]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.154510 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 697e056a55764ed9846cb02442579d03. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.160573 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36a7abd6b4cc4516baf8df29364231e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.160780 1158 tablet_replica.cc:331] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:51:59.160754 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1e87caeaa536423685a228252bb9a901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e87caeaa536423685a228252bb9a901 already in progress: creating tablet
I20250629 01:51:59.160957 1158 raft_consensus.cc:2241] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:59.161146 1158 pending_rounds.cc:70] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:59.161188 1158 pending_rounds.cc:77] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759217517158400 op_type: NO_OP noop_request { }
I20250629 01:51:59.161257 1158 raft_consensus.cc:2887] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:59.161326 1158 raft_consensus.cc:2270] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.162142 1158 ts_tablet_manager.cc:1905] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.166263 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 2de995f275184ec9a2ac43aa496d9cad (DEFAULT_TABLE table=default.Table_af4707cebd704b4ba5a48ab600ab26f0 [id=e64bbc424a344192a4b02b9eb2d8695f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.166553 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de995f275184ec9a2ac43aa496d9cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.167765 1158 ts_tablet_manager.cc:1918] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:59.168421 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 217545a8120a42808dfa71650ed829e0 (DEFAULT_TABLE table=default.Table_6f92518779264d63a2ed0d8468186644 [id=de6e0ab6513d4f82bab83b4234553393]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.174496 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217545a8120a42808dfa71650ed829e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.174643 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee8d44dc4be462f9c15c8360f53143a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.168994 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 24e9caef62f64a4281789aba8f83386a (DEFAULT_TABLE table=default.Table_20c6268f43c64a5480574ad87d428a7c [id=1b897d7a629e4a26998e86e4f1ebb33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.174876 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e9caef62f64a4281789aba8f83386a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.175346 2433 raft_consensus.cc:491] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.175395 2454 raft_consensus.cc:2747] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.175489 2433 raft_consensus.cc:513] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:59.175630 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1a5d7cc320ca455e87d16a78c5f0407d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a5d7cc320ca455e87d16a78c5f0407d already in progress: creating tablet
W20250629 01:51:59.175807 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1e87caeaa536423685a228252bb9a901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e87caeaa536423685a228252bb9a901 already in progress: creating tablet
I20250629 01:51:59.176002 2433 leader_election.cc:290] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.176358 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "258994dd24804f4aaa5e299436fac9b0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:59.176743 712 leader_election.cc:343] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.179837 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d999aeadae450184a384ab906c4af8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:59.180989 988 consensus_peers.cc:489] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:59.184342 1159 ts_tablet_manager.cc:1428] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20250629 01:51:59.187700 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 198d774683a34021be94b8057a5ae5ff (DEFAULT_TABLE table=default.Table_6aad014bd6104b998f834208ad803fdb [id=98ca76427dba4c60acd108dc144ca668]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.187953 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 198d774683a34021be94b8057a5ae5ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.188380 2433 raft_consensus.cc:491] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.188506 2433 raft_consensus.cc:513] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.188723 1158 log.cc:1199] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ac9d77381f7e4241aa546eaea3077168
I20250629 01:51:59.188949 2433 leader_election.cc:290] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.189179 1158 ts_tablet_manager.cc:1939] T ac9d77381f7e4241aa546eaea3077168 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:59.189610 1158 tablet_replica.cc:331] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:59.189965 1158 raft_consensus.cc:2241] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:59.190222 1158 pending_rounds.cc:70] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:59.195924 1155 tablet_bootstrap.cc:492] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.196046 1155 ts_tablet_manager.cc:1397] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.001s
I20250629 01:51:59.196749 1155 raft_consensus.cc:357] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.196895 1155 raft_consensus.cc:383] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.196962 1155 raft_consensus.cc:738] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.197152 1155 consensus_queue.cc:260] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.201098 1159 tablet_bootstrap.cc:492] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.202257 1159 tablet_bootstrap.cc:654] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:59.203944 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8c3c036d11042159d012053bb2a29b3 consensus state has no leader
I20250629 01:51:59.204162 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b5c32f8431be4eda8c77641c20ad4d1e
I20250629 01:51:59.204700 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_da43c1c1f1c948849cef128c6cbaccfe" } new_table_name: "default.Table_71daf3fef0d74acca9e43c6182e18de7" modify_external_catalogs: true
I20250629 01:51:59.206449 2002 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" instance_seqno: 1751161898172646) as {username='slave'} at 127.0.104.2:46979; Asking this server to re-register.
I20250629 01:51:59.207015 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:51:59.207062 877 tablet_service.cc:1515] Processing DeleteTablet for tablet b5c32f8431be4eda8c77641c20ad4d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.207247 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:59.207563 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:59.207762 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:51:59.207926 450 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 013ab393a0f14bdc89389349bcc5d24b (table default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb]). Expected version 1 got 0
W20250629 01:51:59.210666 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b5c32f8431be4eda8c77641c20ad4d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5c32f8431be4eda8c77641c20ad4d1e already in progress: creating tablet
I20250629 01:51:59.213701 750 tablet_service.cc:1515] Processing DeleteTablet for tablet b5c32f8431be4eda8c77641c20ad4d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.221153 963 heartbeater.cc:461] Registering TS with master...
I20250629 01:51:59.235206 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5e8950b2ec654796865087692160d831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.235653 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e8950b2ec654796865087692160d831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.240949 2002 ts_manager.cc:194] Registered new tserver with Master: 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.244606 1159 tablet_bootstrap.cc:492] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.244716 1159 ts_tablet_manager.cc:1397] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20250629 01:51:59.245450 1159 raft_consensus.cc:357] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.245597 1159 raft_consensus.cc:383] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.245649 1159 raft_consensus.cc:738] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.245821 1159 consensus_queue.cc:260] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.260710 1158 pending_rounds.cc:77] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759217138454528 op_type: NO_OP noop_request { }
I20250629 01:51:59.260845 1158 raft_consensus.cc:2887] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:59.260962 1158 raft_consensus.cc:2270] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.262374 1158 ts_tablet_manager.cc:1905] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.266963 1159 ts_tablet_manager.cc:1428] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20250629 01:51:59.174276 1154 tablet_bootstrap.cc:492] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:59.192823 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8 (DEFAULT_TABLE table=default.Table_44f5de58c2f14c7b8c1981d2f735bccd [id=f18fa87c934c430d900e532318700a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.267365 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.268330 1154 tablet_bootstrap.cc:654] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.193830 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 15aef4e26b474f37a4a42e7a7cdf68a1 (DEFAULT_TABLE table=default.Table_aa685973eace4bf5ab1a4d4cbaef51c1 [id=8459cc3f8a5f40f6999b9782d78ffcc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.213742 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 1a5d7cc320ca455e87d16a78c5f0407d (DEFAULT_TABLE table=default.Table_4622cfe68f7a49a8a6d84d12a686dea7 [id=84672e4ccf754d5d936063ca58a7b807]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.214057 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9bbff29d38bf437da3c8273ba1bbec3f (DEFAULT_TABLE table=default.Table_c0c3a2b64e794d63a3c9bce071cf5b97 [id=8a885d9b704b427f9426513b090967dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.214092 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 43b33ec0ed684ae99e49878dc7acbb33 (DEFAULT_TABLE table=default.Table_40a12cf7daec42ef90b22c45964e56f6 [id=527140d11ad841888dbe9466e8e9c2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.214319 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 332a2808e00c4c52afa3931cb8c53516 (DEFAULT_TABLE table=default.Table_a3f4932fdc864c30a60546170db190a9 [id=e0b13eacb71042c8b5f0c8cf8a69c056]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.272657 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a5d7cc320ca455e87d16a78c5f0407d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.272907 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332a2808e00c4c52afa3931cb8c53516. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.214540 1022 tablet_service.cc:1468] Processing CreateTablet for tablet df692a03604444f18cf7b0b337ea3fa5 (DEFAULT_TABLE table=default.Table_1382661b867943bc8431964543552b18 [id=8cf88e839b2c4a969090f8278c34a31e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.276249 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15aef4e26b474f37a4a42e7a7cdf68a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.214845 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 1a5d7cc320ca455e87d16a78c5f0407d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.214907 1015 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d7afa4be82a4ae3974bd0b4786d22db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215133 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 with delete_type TABLET_DATA_DELETED (replaced by tablet b936c61fbbc841278d13fbc9d9860eed) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215353 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet cc65674750c64114b999e0e292a851f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215417 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e87caeaa536423685a228252bb9a901 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:58 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215639 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f3bdc8bda8a940be865679a05b90bd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215747 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215878 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet b5c32f8431be4eda8c77641c20ad4d1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.215919 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8975c1deaaf44761be791f460b53ec74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.276754 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbff29d38bf437da3c8273ba1bbec3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.216609 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.218751 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3a81e6f9711c4d398ae09f33a75566ae" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:59.219120 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d9ef77181ba48f3a6de16b945c8df1a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:59.277026 1047 raft_consensus.cc:2466] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:59.220393 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:59.277192 1052 raft_consensus.cc:3058] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.230500 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 5ec472cd71f04f21aa47b1dfbb32288d (DEFAULT_TABLE table=default.Table_aa8289c495a049b59d261595709a2994 [id=2f803c6780364ab285fa7db60dd77095]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.247664 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "40592f11f430469f9c8e82eba25b38ed" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:59.277575 1044 raft_consensus.cc:3058] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.278005 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ec472cd71f04f21aa47b1dfbb32288d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.279476 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df692a03604444f18cf7b0b337ea3fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.169009 1031 tablet_service.cc:1468] Processing CreateTablet for tablet a2df4e3918614ff0a6644d56c666d764 (DEFAULT_TABLE table=default.Table_57567ad8836042d783d3e6be3438923e [id=80320f4558b644c0b49889bcf23c3e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.281713 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2df4e3918614ff0a6644d56c666d764. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.282517 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b33ec0ed684ae99e49878dc7acbb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.286147 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e3b94f8250a143b096703f2b0386e3cd (table default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]) successfully deleted
W20250629 01:51:59.286482 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f2336702c5e941ce994df6edb1970192 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2336702c5e941ce994df6edb1970192 already in progress: creating tablet
I20250629 01:51:59.286652 1159 tablet_bootstrap.cc:492] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:51:59.286744 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 45461dc364d847d0b0272ac714559008 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45461dc364d847d0b0272ac714559008 already in progress: creating tablet
W20250629 01:51:59.286934 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ec0eeb0ef02c416abc5665f058c6bd7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec0eeb0ef02c416abc5665f058c6bd7f already in progress: creating tablet
I20250629 01:51:59.287238 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 6092e19acf974f2dad5a903da72944ba (table default.Table_913feb22c480457e96599a6814b30c60 [id=abbf099fb0aa4b118f76e5c48700df39]) successfully deleted
I20250629 01:51:59.287261 1052 raft_consensus.cc:2466] T 2ad265c77d0047c38430242c53182e01 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:59.287894 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ac9d77381f7e4241aa546eaea3077168 (table default.Table_92526d1aecd94064817e09aca50dd8d0 [id=ceaecddcc74f48b4ad47d03ea91dd9d6]) successfully deleted
I20250629 01:51:59.287973 1159 tablet_bootstrap.cc:654] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.290455 1044 raft_consensus.cc:2466] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:59.298885 1043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819ed44d98f14075b9a67b88dc44e86e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:59.299046 1043 raft_consensus.cc:2466] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:51:59.299815 710 leader_election.cc:343] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 6092e19acf974f2dad5a903da72944ba
I20250629 01:51:59.300212 710 leader_election.cc:304] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:51:59.300683 710 leader_election.cc:343] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e3b94f8250a143b096703f2b0386e3cd
I20250629 01:51:59.300777 710 leader_election.cc:304] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:59.301079 710 leader_election.cc:304] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:59.301365 710 leader_election.cc:304] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.301996 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1100 } } ignore_live_leader: false dest_uuid: "a0b76160e22c46929936cb9682322fae" is_pre_election: true
I20250629 01:51:59.302109 2049 raft_consensus.cc:2747] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.302547 2433 raft_consensus.cc:2747] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.302569 2049 raft_consensus.cc:2747] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.302722 2049 raft_consensus.cc:2802] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.302786 2049 raft_consensus.cc:491] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.302855 2049 raft_consensus.cc:3058] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.303758 2002 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4712643b602e4f898752fcab6a1e5913" instance_seqno: 1751161898562798) as {username='slave'} at 127.0.104.3:40709; Asking this server to re-register.
I20250629 01:51:59.305428 2049 raft_consensus.cc:513] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.305975 2049 leader_election.cc:290] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.307976 1099 heartbeater.cc:461] Registering TS with master...
W20250629 01:51:59.308780 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 120505us
I20250629 01:51:59.309755 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819ed44d98f14075b9a67b88dc44e86e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:59.312464 2002 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" instance_seqno: 1751161897763572) as {username='slave'} at 127.0.104.1:41577; Asking this server to re-register.
I20250629 01:51:59.312860 1159 tablet_bootstrap.cc:492] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.312976 1159 ts_tablet_manager.cc:1397] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:51:59.313649 1159 raft_consensus.cc:357] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.313788 1159 raft_consensus.cc:383] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.313843 1159 raft_consensus.cc:738] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.314019 1159 consensus_queue.cc:260] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:51:59.314246 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b5c32f8431be4eda8c77641c20ad4d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5c32f8431be4eda8c77641c20ad4d1e already in progress: creating tablet
I20250629 01:51:59.314903 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d9ef77181ba48f3a6de16b945c8df1a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.316495 1154 tablet_bootstrap.cc:492] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:59.316665 1154 ts_tablet_manager.cc:1397] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.003s
W20250629 01:51:59.316709 986 consensus_peers.cc:489] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 -> Peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Couldn't send request to peer ef5c5979bb9e475f983af7c3e9e0e019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68cedbd093b4420d8f7b29219ce8f77b. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:59.316829 1159 ts_tablet_manager.cc:1428] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:51:59.317368 1154 raft_consensus.cc:357] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.317562 1154 raft_consensus.cc:383] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.317654 1154 raft_consensus.cc:738] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.317888 1154 consensus_queue.cc:260] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.318488 1154 ts_tablet_manager.cc:1428] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:59.318792 1154 tablet_bootstrap.cc:492] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:59.319216 824 heartbeater.cc:461] Registering TS with master...
W20250629 01:51:59.322296 712 leader_election.cc:343] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:59.322978 712 leader_election.cc:343] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.324932 2002 ts_manager.cc:194] Registered new tserver with Master: ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:59.326331 710 leader_election.cc:304] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.327257 2049 raft_consensus.cc:2802] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.327368 2049 raft_consensus.cc:695] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:59.327389 710 leader_election.cc:304] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.327579 2049 consensus_queue.cc:237] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.328189 2533 raft_consensus.cc:2802] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.328449 2533 raft_consensus.cc:695] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:59.328691 2533 consensus_queue.cc:237] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.338670 1159 tablet_bootstrap.cc:492] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.340056 1159 tablet_bootstrap.cc:654] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:59.341387 710 leader_election.cc:343] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 258994dd24804f4aaa5e299436fac9b0
I20250629 01:51:59.343107 710 leader_election.cc:304] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:51:59.343497 710 leader_election.cc:304] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.345163 2049 raft_consensus.cc:2802] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.343677 2533 raft_consensus.cc:2747] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.345338 2049 raft_consensus.cc:491] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.345595 2049 raft_consensus.cc:3058] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.348230 2049 raft_consensus.cc:513] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.348481 1155 ts_tablet_manager.cc:1428] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.152s user 0.011s sys 0.009s
I20250629 01:51:59.349237 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819ed44d98f14075b9a67b88dc44e86e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.350243 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_ca0166de4e164756a7d5c1c2972df9a7" } new_table_name: "default.Table_5ae2ada98ea943a4bd45f7e5b21dd961" modify_external_catalogs: true
I20250629 01:51:59.350673 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 7aef2514a87c4c4aaf4d51b314630185 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:59.351629 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 476a6cf28db04f67bc23ddfc5cacad87 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:59.352144 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ec52356e4f4045829090d362590a9ca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.352716 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec52356e4f4045829090d362590a9ca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.354131 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_702cf8b71dad4ba9a893b84bf903252b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.354583 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_702cf8b71dad4ba9a893b84bf903252b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.352188 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_af895bd9f36e4ed4b365f6d299c7f249"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.356395 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af895bd9f36e4ed4b365f6d299c7f249 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.365310 1155 tablet_bootstrap.cc:492] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.367094 1155 tablet_bootstrap.cc:654] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:59.354669 712 leader_election.cc:343] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.383806 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_20ba093e340e4fe7ae570398e255a99b [id=eeb89fb29ca74bd89184ad0a5e2d8c8b] alter complete (version 2)
I20250629 01:51:59.384744 455 master_service.cc:948] ReplaceTablet: received request to replace tablet f7e6f33c7bee44d1a2b6feb0625faedf from {username='slave'} at 127.0.0.1:56238
W20250629 01:51:59.384961 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e3ecff91fcf4fa080f9c4d80a2c9ee8 consensus state has no leader
I20250629 01:51:59.385279 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20c6268f43c64a5480574ad87d428a7c" } modify_external_catalogs: true
I20250629 01:51:59.348882 2049 leader_election.cc:290] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.385418 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20c6268f43c64a5480574ad87d428a7c" } modify_external_catalogs: true
I20250629 01:51:59.386056 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6428a3757bd44307b68b33e17a0eec05" } modify_external_catalogs: true
I20250629 01:51:59.386178 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6428a3757bd44307b68b33e17a0eec05" } modify_external_catalogs: true
W20250629 01:51:59.388269 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2ce14bc48f24499ad1a02fbd53fbe08 consensus state has no leader
I20250629 01:51:59.388523 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_910f8f6757c44909a04eb629bfa037e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.388866 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_910f8f6757c44909a04eb629bfa037e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.390050 1159 tablet_bootstrap.cc:492] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.390143 1159 ts_tablet_manager.cc:1397] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.003s
I20250629 01:51:59.390705 1159 raft_consensus.cc:357] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.390825 1159 raft_consensus.cc:383] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.390861 1159 raft_consensus.cc:738] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.391018 1159 consensus_queue.cc:260] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.391429 1159 ts_tablet_manager.cc:1428] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:59.391650 1159 tablet_bootstrap.cc:492] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.393381 1501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1100 } } ignore_live_leader: false dest_uuid: "f304077528b34c61857a659e5873631d" is_pre_election: true
I20250629 01:51:59.394084 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 3db333c60cb3485a9daae316b9f1a8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.394297 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 54f38ccc485043ba8b8a6f644a3e4f36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.394917 2526 tablet_replica.cc:331] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.395071 2526 raft_consensus.cc:2241] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.395016 1969 leader_election.cc:304] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f4258c004e04bcf8216bb7c8fd56c54; no voters: a0b76160e22c46929936cb9682322fae, f304077528b34c61857a659e5873631d
I20250629 01:51:59.395179 2526 raft_consensus.cc:2270] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.395465 2466 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:51:59.395617 2526 ts_tablet_manager.cc:1905] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.398926 750 tablet_service.cc:1515] Processing DeleteTablet for tablet fbcae02d893f4795a12d26af57b18b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.399135 1156 tablet_replica.cc:331] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.399308 1156 raft_consensus.cc:2241] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.399454 1156 raft_consensus.cc:2270] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.399909 1156 ts_tablet_manager.cc:1905] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.402827 2526 ts_tablet_manager.cc:1918] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.405858 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:59.406344 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:59.406610 1156 ts_tablet_manager.cc:1918] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.406730 1156 log.cc:1199] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/fbcae02d893f4795a12d26af57b18b85
I20250629 01:51:59.406661 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943]). Expected version 5 got 0
I20250629 01:51:59.407020 1156 ts_tablet_manager.cc:1939] T fbcae02d893f4795a12d26af57b18b85 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.407869 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_825efeee6983439b9a45a82092f1e29e" } modify_external_catalogs: true
I20250629 01:51:59.408041 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_825efeee6983439b9a45a82092f1e29e" } modify_external_catalogs: true
I20250629 01:51:59.404021 1159 tablet_bootstrap.cc:654] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.409404 2526 log.cc:1199] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/3db333c60cb3485a9daae316b9f1a8c8
I20250629 01:51:59.409984 2526 ts_tablet_manager.cc:1939] T 3db333c60cb3485a9daae316b9f1a8c8 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:59.410400 2526 tablet_replica.cc:331] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.410784 2526 raft_consensus.cc:2241] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.411144 2526 raft_consensus.cc:2270] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.412284 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 3db333c60cb3485a9daae316b9f1a8c8 (table default.Table_f81f54577f464e6bbe68673617d0e0cd [id=d929c72226424c82a6d8c8bac1c3aac0]) successfully deleted
I20250629 01:51:59.412796 2526 ts_tablet_manager.cc:1905] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.415019 2049 raft_consensus.cc:491] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.415189 2049 raft_consensus.cc:513] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.415746 2049 leader_election.cc:290] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.416040 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3bdc8bda8a940be865679a05b90bd08" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:59.416543 712 leader_election.cc:343] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.416723 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9d77381f7e4241aa546eaea3077168 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.417007 1156 tablet_replica.cc:331] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.417217 1156 raft_consensus.cc:2241] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.417424 1156 raft_consensus.cc:2270] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.417883 1156 ts_tablet_manager.cc:1905] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.420212 2454 raft_consensus.cc:491] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.420344 2454 raft_consensus.cc:513] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.420909 2454 leader_election.cc:290] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:59.421635 850 leader_election.cc:343] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a6a012ba891949538800581436ef5380
W20250629 01:51:59.421802 850 leader_election.cc:343] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: a6a012ba891949538800581436ef5380
I20250629 01:51:59.421875 850 leader_election.cc:304] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.318949 2527 raft_consensus.cc:491] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.422044 2454 raft_consensus.cc:2747] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.350984 1043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d9ef77181ba48f3a6de16b945c8df1a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:59.422142 2527 raft_consensus.cc:513] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.361173 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "819ed44d98f14075b9a67b88dc44e86e" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:59.422317 1044 raft_consensus.cc:3058] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.422740 2527 leader_election.cc:290] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.423544 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b8a2d1a7f7146f4968590914374ba23" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:51:59.423899 986 leader_election.cc:343] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.379194 2562 tablet.cc:1722] T 8f9f59fbe9634ac8b27f990d9496f637 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.424619 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b8a2d1a7f7146f4968590914374ba23" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:59.424757 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6092e19acf974f2dad5a903da72944ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:51:59.424997 988 leader_election.cc:343] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.425096 988 leader_election.cc:304] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.382154 2561 tablet.cc:1722] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.417896 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3bdc8bda8a940be865679a05b90bd08" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:51:59.422210 1043 raft_consensus.cc:3058] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.426213 1052 raft_consensus.cc:2466] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:51:59.427955 710 leader_election.cc:304] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.428176 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e31520d3ad4c3395a59049755a4725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.428315 2049 raft_consensus.cc:2802] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.428428 2049 raft_consensus.cc:491] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.428531 2049 raft_consensus.cc:3058] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.430660 2049 raft_consensus.cc:513] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.431632 2049 leader_election.cc:290] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.369390 1158 ts_tablet_manager.cc:1918] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:59.432329 1158 log.cc:1199] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/fbcae02d893f4795a12d26af57b18b85
I20250629 01:51:59.432747 1158 ts_tablet_manager.cc:1939] T fbcae02d893f4795a12d26af57b18b85 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:59.433640 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 258994dd24804f4aaa5e299436fac9b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.434266 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet fbcae02d893f4795a12d26af57b18b85 (table default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 [id=a128a46807814758b9b79071ca6db923]) successfully deleted
I20250629 01:51:59.434240 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3bdc8bda8a940be865679a05b90bd08" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:59.434810 712 leader_election.cc:343] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.437840 2561 tablet.cc:1722] T 1bc3d470df0d4814a655bcff68c956e2 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:51:59.438999 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c84dfb60f67548ec9fa4c84c42e500c8 consensus state has no leader
I20250629 01:51:59.442781 452 catalog_manager.cc:5582] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:51:59.443280 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:59.443423 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:51:59.443785 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
I20250629 01:51:59.444144 1155 tablet_bootstrap.cc:492] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.444258 1155 ts_tablet_manager.cc:1397] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.004s
I20250629 01:51:59.444839 1155 raft_consensus.cc:357] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.444972 1155 raft_consensus.cc:383] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.445034 1155 raft_consensus.cc:738] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.445199 1155 consensus_queue.cc:260] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.446156 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d74571bf53c54b24aa34535f695efa89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.446544 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d74571bf53c54b24aa34535f695efa89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.447525 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 12d63924c5ff4a99ac62f89a9d29423b from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:59.449587 2454 raft_consensus.cc:491] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.449710 2454 raft_consensus.cc:513] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.450033 892 tablet_service.cc:1468] Processing CreateTablet for tablet de8af6b0b82840fd86658e72523c80ce (DEFAULT_TABLE table=default.Table_7a293230f8fb43b0aab93a9e76852915 [id=dbe540c577a749b0bd535a4cf52b4c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.450140 2454 leader_election.cc:290] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:59.450335 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8af6b0b82840fd86658e72523c80ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.450388 890 tablet_service.cc:1468] Processing CreateTablet for tablet c0e119e1819544dc9eb5f6fe3823c612 (DEFAULT_TABLE table=default.Table_84c1f44d16dd4a46b714a94ea498d75d [id=50fd12e423d74e279d966f73571997b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.450639 877 tablet_service.cc:1468] Processing CreateTablet for tablet d4ed1a71a32644819adf83ebbadaf895 (DEFAULT_TABLE table=default.Table_708d2b9c3b804e6288ce7b9226d76bac [id=cc7447b71c5443dcae2e0632a6c52b16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.450857 896 tablet_service.cc:1468] Processing CreateTablet for tablet 02f3035ce2c949e5836bcfba74919e83 (DEFAULT_TABLE table=default.Table_f3b9149e73b24dac8af5e1d055f3f63d [id=ae530f8a7bcf4fa38da012f13f8dd7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.450652 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e119e1819544dc9eb5f6fe3823c612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.451534 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ed1a71a32644819adf83ebbadaf895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.452291 880 tablet_service.cc:1468] Processing CreateTablet for tablet 85cb58bbde634778aab9ab0c72d0f276 (DEFAULT_TABLE table=default.Table_34d94473b0b1431ba7da4d005a8f3b91 [id=36c588bed83e4ca1a5a6e39e18ee51e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.452531 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85cb58bbde634778aab9ab0c72d0f276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.455447 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f3035ce2c949e5836bcfba74919e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.456284 886 tablet_service.cc:1468] Processing CreateTablet for tablet 849e3e59b9624994bbb63027aa85d0b0 (DEFAULT_TABLE table=default.Table_d6fad403f1ae4427895f31b28a95daf3 [id=d271d5ee6c9b4ef180fad3fc21c7425a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.456532 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849e3e59b9624994bbb63027aa85d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.459414 750 tablet_service.cc:1468] Processing CreateTablet for tablet d4ed1a71a32644819adf83ebbadaf895 (DEFAULT_TABLE table=default.Table_708d2b9c3b804e6288ce7b9226d76bac [id=cc7447b71c5443dcae2e0632a6c52b16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.459776 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ed1a71a32644819adf83ebbadaf895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.460294 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2aba500fce284ad8b020e60a0c0f7741 (DEFAULT_TABLE table=default.Table_1ece6af955a54ec2b5819ceba5d2ec17 [id=85897178e43846b7b9443c9cbe78d2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.460397 1155 ts_tablet_manager.cc:1428] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250629 01:51:59.460565 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aba500fce284ad8b020e60a0c0f7741. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.462203 1155 tablet_bootstrap.cc:492] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.464367 885 tablet_service.cc:1468] Processing CreateTablet for tablet b936c61fbbc841278d13fbc9d9860eed (DEFAULT_TABLE table=default.Table_0cc47fb2736e4c40ba44dae6f0dec98d [id=ece0dac89dd34051bd2f1163dd3c9bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.464643 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b936c61fbbc841278d13fbc9d9860eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.465212 2558 tablet.cc:1722] T 8f9f59fbe9634ac8b27f990d9496f637 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.465915 2572 tablet.cc:1722] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.466126 750 tablet_service.cc:1468] Processing CreateTablet for tablet de8af6b0b82840fd86658e72523c80ce (DEFAULT_TABLE table=default.Table_7a293230f8fb43b0aab93a9e76852915 [id=dbe540c577a749b0bd535a4cf52b4c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.466405 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8af6b0b82840fd86658e72523c80ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:59.456482 850 leader_election.cc:343] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 52be184d0480495a894442d6f72a0f12
I20250629 01:51:59.474298 1044 raft_consensus.cc:2466] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:59.474822 710 leader_election.cc:304] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.475086 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3bdc8bda8a940be865679a05b90bd08" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:51:59.475242 1044 raft_consensus.cc:3058] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.475492 2553 raft_consensus.cc:2802] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.475875 2553 raft_consensus.cc:695] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:59.476204 2553 consensus_queue.cc:237] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.477026 1013 tablet_service.cc:1468] Processing CreateTablet for tablet de8af6b0b82840fd86658e72523c80ce (DEFAULT_TABLE table=default.Table_7a293230f8fb43b0aab93a9e76852915 [id=dbe540c577a749b0bd535a4cf52b4c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.477392 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de8af6b0b82840fd86658e72523c80ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.478667 1031 tablet_service.cc:1468] Processing CreateTablet for tablet d4ed1a71a32644819adf83ebbadaf895 (DEFAULT_TABLE table=default.Table_708d2b9c3b804e6288ce7b9226d76bac [id=cc7447b71c5443dcae2e0632a6c52b16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.478983 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ed1a71a32644819adf83ebbadaf895. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.478983 1155 tablet_bootstrap.cc:654] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.479984 757 tablet_service.cc:1468] Processing CreateTablet for tablet c0e119e1819544dc9eb5f6fe3823c612 (DEFAULT_TABLE table=default.Table_84c1f44d16dd4a46b714a94ea498d75d [id=50fd12e423d74e279d966f73571997b4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:59.480436 850 leader_election.cc:343] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 52be184d0480495a894442d6f72a0f12
I20250629 01:51:59.480532 850 leader_election.cc:304] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.480835 747 tablet_service.cc:1468] Processing CreateTablet for tablet 02f3035ce2c949e5836bcfba74919e83 (DEFAULT_TABLE table=default.Table_f3b9149e73b24dac8af5e1d055f3f63d [id=ae530f8a7bcf4fa38da012f13f8dd7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.481144 2454 raft_consensus.cc:2747] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.481148 746 tablet_service.cc:1468] Processing CreateTablet for tablet 85cb58bbde634778aab9ab0c72d0f276 (DEFAULT_TABLE table=default.Table_34d94473b0b1431ba7da4d005a8f3b91 [id=36c588bed83e4ca1a5a6e39e18ee51e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.481555 1158 tablet_replica.cc:331] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:59.482478 742 tablet_service.cc:1468] Processing CreateTablet for tablet 849e3e59b9624994bbb63027aa85d0b0 (DEFAULT_TABLE table=default.Table_d6fad403f1ae4427895f31b28a95daf3 [id=d271d5ee6c9b4ef180fad3fc21c7425a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.483883 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2aba500fce284ad8b020e60a0c0f7741 (DEFAULT_TABLE table=default.Table_1ece6af955a54ec2b5819ceba5d2ec17 [id=85897178e43846b7b9443c9cbe78d2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.484756 752 tablet_service.cc:1468] Processing CreateTablet for tablet b936c61fbbc841278d13fbc9d9860eed (DEFAULT_TABLE table=default.Table_0cc47fb2736e4c40ba44dae6f0dec98d [id=ece0dac89dd34051bd2f1163dd3c9bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.484741 1021 tablet_service.cc:1468] Processing CreateTablet for tablet c0e119e1819544dc9eb5f6fe3823c612 (DEFAULT_TABLE table=default.Table_84c1f44d16dd4a46b714a94ea498d75d [id=50fd12e423d74e279d966f73571997b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.484922 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 85cb58bbde634778aab9ab0c72d0f276 (DEFAULT_TABLE table=default.Table_34d94473b0b1431ba7da4d005a8f3b91 [id=36c588bed83e4ca1a5a6e39e18ee51e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.485029 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e119e1819544dc9eb5f6fe3823c612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.485208 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85cb58bbde634778aab9ab0c72d0f276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.485280 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 02f3035ce2c949e5836bcfba74919e83 (DEFAULT_TABLE table=default.Table_f3b9149e73b24dac8af5e1d055f3f63d [id=ae530f8a7bcf4fa38da012f13f8dd7d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.485714 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4ec7b96fd1dc4d2c81ad425489a1cdee (DEFAULT_TABLE table=default.Table_6fda51d9864f43b6ae070bb2a165ebad [id=620ef91f672c4e4e90b36d4bbdaaba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.486020 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec7b96fd1dc4d2c81ad425489a1cdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.490135 2487 tablet.cc:1722] T 5860f7100a78476b8fa1d34f6c0b8d77 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:59.490479 886 tablet_service.cc:1468] Processing CreateTablet for tablet a1da6e7dee904f23886f6d9d75c8e49f (DEFAULT_TABLE table=default.Table_a45ac1d6a41941178905d51452d4c4fc [id=6efdb62e7ce9414e9a0dbcf1c968f28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.490764 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1da6e7dee904f23886f6d9d75c8e49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.492309 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 849e3e59b9624994bbb63027aa85d0b0 (DEFAULT_TABLE table=default.Table_d6fad403f1ae4427895f31b28a95daf3 [id=d271d5ee6c9b4ef180fad3fc21c7425a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.492588 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849e3e59b9624994bbb63027aa85d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.494009 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f3035ce2c949e5836bcfba74919e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.496304 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 2aba500fce284ad8b020e60a0c0f7741 (DEFAULT_TABLE table=default.Table_1ece6af955a54ec2b5819ceba5d2ec17 [id=85897178e43846b7b9443c9cbe78d2a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.496551 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aba500fce284ad8b020e60a0c0f7741. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.498237 2572 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:38530 (request call id 1244) took 1290 ms. Trace:
I20250629 01:51:59.498741 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b936c61fbbc841278d13fbc9d9860eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.498322 2572 rpcz_store.cc:276] 0629 01:51:58.207439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.207482 (+ 43us) service_pool.cc:225] Handling call
0629 01:51:59.498201 (+1290719us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0629 01:51:58.207772 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0629 01:51:58.207873 (+ 101us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0629 01:51:58.207880 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0629 01:51:58.207886 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0629 01:51:58.207918 (+ 32us) alter_schema_op.cc:128] START. Timestamp: 1751161918207802
0629 01:51:58.207922 (+ 4us) op_driver.cc:348] REPLICATION: starting
0629 01:51:58.208111 (+ 189us) log.cc:844] Serialized 173 byte log entry
0629 01:51:59.435052 (+1226941us) op_driver.cc:464] REPLICATION: finished
0629 01:51:59.465878 (+ 30826us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0629 01:51:59.466214 (+ 336us) log.cc:844] Serialized 5 byte log entry
0629 01:51:59.497962 (+ 31748us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.498142 (+ 180us) log.cc:844] Serialized 14 byte log entry
0629 01:51:59.498171 (+ 29us) alter_schema_op.cc:73] Released schema lock
0629 01:51:59.498175 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30776,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":412,"prepare.run_wall_time_us":413,"raft.queue_time_us":115,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"replication_time_us":1227100,"thread_start_us":31181,"threads_started":1}]]}
I20250629 01:51:59.499058 2572 tablet.cc:1722] T 5860f7100a78476b8fa1d34f6c0b8d77 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:59.499157 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 849e3e59b9624994bbb63027aa85d0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.499392 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02f3035ce2c949e5836bcfba74919e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.502341 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85cb58bbde634778aab9ab0c72d0f276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.507553 884 tablet_service.cc:1468] Processing CreateTablet for tablet 2d106ea1073e45feb02fc215aa4b5d51 (DEFAULT_TABLE table=default.Table_8844355e9b1144cd9cb3b2787a80e01c [id=3fe12c402e8c44fdaaeeb691f4673731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.507648 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2aba500fce284ad8b020e60a0c0f7741. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.508025 2454 raft_consensus.cc:491] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.508162 2454 raft_consensus.cc:513] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.508334 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d106ea1073e45feb02fc215aa4b5d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.508666 2454 leader_election.cc:290] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:59.512511 1155 tablet_bootstrap.cc:492] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.512684 1155 ts_tablet_manager.cc:1397] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.001s
I20250629 01:51:59.512743 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0e119e1819544dc9eb5f6fe3823c612. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:59.512743 850 leader_election.cc:343] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: da93c730451d42bc8df1e582fb3af818
I20250629 01:51:59.513368 1155 raft_consensus.cc:357] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.513509 1155 raft_consensus.cc:383] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.513564 1155 raft_consensus.cc:738] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.513757 1155 consensus_queue.cc:260] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.514166 1155 ts_tablet_manager.cc:1428] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:59.514407 1155 tablet_bootstrap.cc:492] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.524657 2577 tablet.cc:1722] T 347fa6a12a744ee88058643b7143e39d P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:59.530489 2577 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:38530 (request call id 1211) took 1686 ms. Trace:
I20250629 01:51:59.530632 2577 rpcz_store.cc:276] 0629 01:51:57.843850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:57.844386 (+ 536us) service_pool.cc:225] Handling call
0629 01:51:59.530459 (+1686073us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0629 01:51:57.844698 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0629 01:51:57.844779 (+ 81us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0629 01:51:57.844786 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
0629 01:51:57.844791 (+ 5us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0629 01:51:57.844817 (+ 26us) alter_schema_op.cc:128] START. Timestamp: 1751161917844779
0629 01:51:57.844819 (+ 2us) op_driver.cc:348] REPLICATION: starting
0629 01:51:57.846000 (+ 1181us) log.cc:844] Serialized 173 byte log entry
0629 01:51:59.523117 (+1677117us) op_driver.cc:464] REPLICATION: finished
0629 01:51:59.524619 (+ 1502us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0629 01:51:59.525891 (+ 1272us) log.cc:844] Serialized 5 byte log entry
0629 01:51:59.530103 (+ 4212us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.530272 (+ 169us) log.cc:844] Serialized 14 byte log entry
0629 01:51:59.530435 (+ 163us) alter_schema_op.cc:73] Released schema lock
0629 01:51:59.530439 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1452,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":380,"prepare.run_wall_time_us":1394,"raft.queue_time_us":271,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":236,"replication_time_us":1678266}]]}
I20250629 01:51:59.533296 1155 tablet_bootstrap.cc:654] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.540351 2570 raft_consensus.cc:2747] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.540314 2561 tablet.cc:1722] T 347fa6a12a744ee88058643b7143e39d P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:51:59.540506 1155 tablet_bootstrap.cc:492] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.540609 1155 ts_tablet_manager.cc:1397] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:51:59.541205 1155 raft_consensus.cc:357] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.541356 1155 raft_consensus.cc:383] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.541419 1155 raft_consensus.cc:738] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.541458 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2d106ea1073e45feb02fc215aa4b5d51 (DEFAULT_TABLE table=default.Table_8844355e9b1144cd9cb3b2787a80e01c [id=3fe12c402e8c44fdaaeeb691f4673731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.541638 1155 consensus_queue.cc:260] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.541687 742 tablet_service.cc:1468] Processing CreateTablet for tablet a1da6e7dee904f23886f6d9d75c8e49f (DEFAULT_TABLE table=default.Table_a45ac1d6a41941178905d51452d4c4fc [id=6efdb62e7ce9414e9a0dbcf1c968f28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.541772 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d106ea1073e45feb02fc215aa4b5d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.541950 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1da6e7dee904f23886f6d9d75c8e49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.542093 1155 ts_tablet_manager.cc:1428] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:59.542171 752 tablet_service.cc:1468] Processing CreateTablet for tablet 4ec7b96fd1dc4d2c81ad425489a1cdee (DEFAULT_TABLE table=default.Table_6fda51d9864f43b6ae070bb2a165ebad [id=620ef91f672c4e4e90b36d4bbdaaba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.542354 1155 tablet_bootstrap.cc:492] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.542856 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2d106ea1073e45feb02fc215aa4b5d51 (DEFAULT_TABLE table=default.Table_8844355e9b1144cd9cb3b2787a80e01c [id=3fe12c402e8c44fdaaeeb691f4673731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.543143 1044 raft_consensus.cc:2466] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:59.543146 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d106ea1073e45feb02fc215aa4b5d51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.546219 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a1da6e7dee904f23886f6d9d75c8e49f (DEFAULT_TABLE table=default.Table_a45ac1d6a41941178905d51452d4c4fc [id=6efdb62e7ce9414e9a0dbcf1c968f28a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.546515 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1da6e7dee904f23886f6d9d75c8e49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.547075 710 leader_election.cc:304] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.549351 1155 tablet_bootstrap.cc:654] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.549849 2548 raft_consensus.cc:2802] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.549979 2548 raft_consensus.cc:695] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:59.550200 2548 consensus_queue.cc:237] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.552872 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 4ec7b96fd1dc4d2c81ad425489a1cdee (DEFAULT_TABLE table=default.Table_6fda51d9864f43b6ae070bb2a165ebad [id=620ef91f672c4e4e90b36d4bbdaaba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.553125 2526 ts_tablet_manager.cc:1918] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.553155 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec7b96fd1dc4d2c81ad425489a1cdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.553224 2526 log.cc:1199] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/54f38ccc485043ba8b8a6f644a3e4f36
I20250629 01:51:59.553460 2526 ts_tablet_manager.cc:1939] T 54f38ccc485043ba8b8a6f644a3e4f36 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:59.555423 1018 tablet_service.cc:1468] Processing CreateTablet for tablet b936c61fbbc841278d13fbc9d9860eed (DEFAULT_TABLE table=default.Table_0cc47fb2736e4c40ba44dae6f0dec98d [id=ece0dac89dd34051bd2f1163dd3c9bf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.555752 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b936c61fbbc841278d13fbc9d9860eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.560938 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet fbcae02d893f4795a12d26af57b18b85 (table default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 [id=a128a46807814758b9b79071ca6db923]) successfully deleted
I20250629 01:51:59.561231 1155 tablet_bootstrap.cc:492] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.561589 1155 ts_tablet_manager.cc:1397] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20250629 01:51:59.563320 1159 tablet_bootstrap.cc:492] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.563498 1159 ts_tablet_manager.cc:1397] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.172s user 0.003s sys 0.007s
I20250629 01:51:59.564283 1159 raft_consensus.cc:357] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.564469 1159 raft_consensus.cc:383] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.564574 1159 raft_consensus.cc:738] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.564802 1159 consensus_queue.cc:260] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.565807 1159 ts_tablet_manager.cc:1428] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:51:59.566498 1156 ts_tablet_manager.cc:1918] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.566625 1156 log.cc:1199] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ac9d77381f7e4241aa546eaea3077168
I20250629 01:51:59.566690 1159 tablet_bootstrap.cc:492] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.568545 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 54f38ccc485043ba8b8a6f644a3e4f36 (table default.Table_119622f658584820a8fcd60b2b640e05 [id=c2c6ad07071540d09fb8e8e13d4e5bea]) successfully deleted
I20250629 01:51:59.570147 1159 tablet_bootstrap.cc:654] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.575373 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2f55e3c1a16e4082aaacb9434d6d583c" } modify_external_catalogs: true
I20250629 01:51:59.575525 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2f55e3c1a16e4082aaacb9434d6d583c" } modify_external_catalogs: true
I20250629 01:51:59.576740 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7aef2514a87c4c4aaf4d51b314630185
I20250629 01:51:59.577052 451 catalog_manager.cc:6652] ReplaceTablet: tablet 7aef2514a87c4c4aaf4d51b314630185 deleted and replaced by tablet d4b4e75281ba4cd8a8e1f0a47710964f
I20250629 01:51:59.577247 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9abd434ca0b940478c9661546ed480a2" } modify_external_catalogs: true
I20250629 01:51:59.577364 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9abd434ca0b940478c9661546ed480a2" } modify_external_catalogs: true
I20250629 01:51:59.578361 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ab59d0a03543499fbacaf39743ca812b
I20250629 01:51:59.578841 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_39a0ff1dd8064dc4a2990b95fac35099" } new_table_name: "default.Table_8461ee9ea7fa468dba21bab26f97cd23" modify_external_catalogs: true
I20250629 01:51:59.579762 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f7e6f33c7bee44d1a2b6feb0625faedf
I20250629 01:51:59.580029 455 catalog_manager.cc:6652] ReplaceTablet: tablet f7e6f33c7bee44d1a2b6feb0625faedf deleted and replaced by tablet 2fa581a6d6f64f0eb18f976caf675858
I20250629 01:51:59.580185 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 504c15b3b168499d9a82584f82a0b792 from {username='slave'} at 127.0.0.1:56238
I20250629 01:51:59.581522 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_226a81ddbfc54450b599ac93e64a0517"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.582000 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_226a81ddbfc54450b599ac93e64a0517 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.583125 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 12d63924c5ff4a99ac62f89a9d29423b
I20250629 01:51:59.583472 448 catalog_manager.cc:6652] ReplaceTablet: tablet 12d63924c5ff4a99ac62f89a9d29423b deleted and replaced by tablet f4cdefd680414cbda42e4090e78b1fd2
I20250629 01:51:59.583699 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9f48c56326b54e74ac850d12a81563cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.584084 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f48c56326b54e74ac850d12a81563cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.584139 1156 ts_tablet_manager.cc:1939] T ac9d77381f7e4241aa546eaea3077168 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.566033 1155 raft_consensus.cc:357] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.584666 1155 raft_consensus.cc:383] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.584721 1155 raft_consensus.cc:738] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.584877 1155 consensus_queue.cc:260] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.585417 1155 ts_tablet_manager.cc:1428] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.024s user 0.000s sys 0.001s
I20250629 01:51:59.585563 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 12d63924c5ff4a99ac62f89a9d29423b with delete_type TABLET_DATA_DELETED (replaced by tablet f4cdefd680414cbda42e4090e78b1fd2) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.585646 1155 tablet_bootstrap.cc:492] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.585845 886 tablet_service.cc:1515] Processing DeleteTablet for tablet f7e6f33c7bee44d1a2b6feb0625faedf with delete_type TABLET_DATA_DELETED (replaced by tablet 2fa581a6d6f64f0eb18f976caf675858) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.586043 880 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59d0a03543499fbacaf39743ca812b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.586236 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 7aef2514a87c4c4aaf4d51b314630185 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b4e75281ba4cd8a8e1f0a47710964f) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.586844 1155 tablet_bootstrap.cc:654] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:59.588935 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 12d63924c5ff4a99ac62f89a9d29423b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12d63924c5ff4a99ac62f89a9d29423b already in progress: creating tablet
W20250629 01:51:59.589114 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f7e6f33c7bee44d1a2b6feb0625faedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7e6f33c7bee44d1a2b6feb0625faedf already in progress: creating tablet
I20250629 01:51:59.589247 1156 tablet_replica.cc:331] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:51:59.589274 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ab59d0a03543499fbacaf39743ca812b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab59d0a03543499fbacaf39743ca812b already in progress: creating tablet
I20250629 01:51:59.589401 1156 raft_consensus.cc:2241] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:51:59.589432 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7aef2514a87c4c4aaf4d51b314630185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7aef2514a87c4c4aaf4d51b314630185 already in progress: creating tablet
I20250629 01:51:59.589511 1156 raft_consensus.cc:2270] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.590003 1156 ts_tablet_manager.cc:1905] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.589756 1159 tablet_bootstrap.cc:492] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.592159 1159 ts_tablet_manager.cc:1397] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250629 01:51:59.592809 1159 raft_consensus.cc:357] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.593087 1159 raft_consensus.cc:383] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.593178 1159 raft_consensus.cc:738] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.593402 1159 consensus_queue.cc:260] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.593799 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ac9d77381f7e4241aa546eaea3077168 (table default.Table_92526d1aecd94064817e09aca50dd8d0 [id=ceaecddcc74f48b4ad47d03ea91dd9d6]) successfully deleted
I20250629 01:51:59.595952 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 7aef2514a87c4c4aaf4d51b314630185 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b4e75281ba4cd8a8e1f0a47710964f) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.596022 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 7aef2514a87c4c4aaf4d51b314630185 with delete_type TABLET_DATA_DELETED (replaced by tablet d4b4e75281ba4cd8a8e1f0a47710964f) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.595999 742 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59d0a03543499fbacaf39743ca812b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.598850 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ab59d0a03543499fbacaf39743ca812b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.600142 1159 ts_tablet_manager.cc:1428] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250629 01:51:59.600510 1159 tablet_bootstrap.cc:492] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.602371 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec7b96fd1dc4d2c81ad425489a1cdee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.604527 1159 tablet_bootstrap.cc:654] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.618582 1154 tablet_bootstrap.cc:654] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.620949 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 12d63924c5ff4a99ac62f89a9d29423b with delete_type TABLET_DATA_DELETED (replaced by tablet f4cdefd680414cbda42e4090e78b1fd2) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.621372 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet f7e6f33c7bee44d1a2b6feb0625faedf with delete_type TABLET_DATA_DELETED (replaced by tablet 2fa581a6d6f64f0eb18f976caf675858) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.609076 742 tablet_service.cc:1515] Processing DeleteTablet for tablet f7e6f33c7bee44d1a2b6feb0625faedf with delete_type TABLET_DATA_DELETED (replaced by tablet 2fa581a6d6f64f0eb18f976caf675858) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.622103 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 46bb76c63ce749078365420f224d62f3
I20250629 01:51:59.622750 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_11fa9f38439b4f6d990266584eebfaf9" } new_table_name: "default.Table_1a19733c02cd4709881a1136ce825e1c" modify_external_catalogs: true
I20250629 01:51:59.624025 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c4c4153b6dcd45ada716c05d0590d10a
I20250629 01:51:59.624059 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 24e9caef62f64a4281789aba8f83386a
W20250629 01:51:59.624380 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 196560us
I20250629 01:51:59.624583 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_36cbb7b4045745cc8a65574c33d2ea65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:59.624697 1156 ts_tablet_manager.cc:1918] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.624768 1156 log.cc:1199] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/6092e19acf974f2dad5a903da72944ba
I20250629 01:51:59.624989 1156 ts_tablet_manager.cc:1939] T 6092e19acf974f2dad5a903da72944ba P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:51:59.624994 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36cbb7b4045745cc8a65574c33d2ea65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.625694 1156 tablet_replica.cc:331] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.625850 1156 raft_consensus.cc:2241] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.625969 1156 raft_consensus.cc:2270] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.625934 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2b15bee7396749e4bf96b95d85662564"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.626364 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b15bee7396749e4bf96b95d85662564 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.626452 1156 ts_tablet_manager.cc:1905] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.627761 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bb76c63ce749078365420f224d62f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.627909 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 24e9caef62f64a4281789aba8f83386a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.628000 886 tablet_service.cc:1515] Processing DeleteTablet for tablet c4c4153b6dcd45ada716c05d0590d10a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:59.629231 850 leader_election.cc:343] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: da93c730451d42bc8df1e582fb3af818
I20250629 01:51:59.629355 850 leader_election.cc:304] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.629568 2454 raft_consensus.cc:2747] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.629556 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:51:59.629904 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:59.630052 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 3b55c66136ba43d7b535750e04b72308 (table default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56]). Expected version 1 got 0
W20250629 01:51:59.632969 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 631ca7239f574441b37c11e104c3d56b consensus state has no leader
I20250629 01:51:59.633261 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d96f54aeed554f3bbe79066de62debc5" } new_table_name: "default.Table_b711a9cf14ee4bebb7cd6f01510807aa" modify_external_catalogs: true
I20250629 01:51:59.633374 886 tablet_service.cc:1515] Processing DeleteTablet for tablet da93c730451d42bc8df1e582fb3af818 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.633527 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 2aca26af31ed4bc994f4d62603edc881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.634017 2526 tablet_replica.cc:331] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.634168 2526 raft_consensus.cc:2241] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.634296 2526 raft_consensus.cc:2270] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.634703 2526 ts_tablet_manager.cc:1905] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:51:59.635087 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 46bb76c63ce749078365420f224d62f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46bb76c63ce749078365420f224d62f3 already in progress: creating tablet
I20250629 01:51:59.635210 2558 tablet.cc:1722] T 1bc3d470df0d4814a655bcff68c956e2 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:51:59.635280 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 24e9caef62f64a4281789aba8f83386a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24e9caef62f64a4281789aba8f83386a already in progress: creating tablet
W20250629 01:51:59.635481 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c4c4153b6dcd45ada716c05d0590d10a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4c4153b6dcd45ada716c05d0590d10a already in progress: creating tablet
I20250629 01:51:59.640314 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 504c15b3b168499d9a82584f82a0b792
I20250629 01:51:59.640719 455 catalog_manager.cc:6652] ReplaceTablet: tablet 504c15b3b168499d9a82584f82a0b792 deleted and replaced by tablet 4c34a9c407304a6e825c61d81bde6ac2
I20250629 01:51:59.640933 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1aba3deb8ac142f9979055dad2418720"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.641265 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1aba3deb8ac142f9979055dad2418720 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.642818 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8ae0d61d254c4e1ba508e30aa35600ca
I20250629 01:51:59.643278 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9a012ad660f649da977b3123b27ecd2e" } new_table_name: "default.Table_75ca31dbf2c745918cc69f2166dd9b89" modify_external_catalogs: true
I20250629 01:51:59.644443 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 504c15b3b168499d9a82584f82a0b792 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c34a9c407304a6e825c61d81bde6ac2) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.644575 2553 raft_consensus.cc:491] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.644670 2553 raft_consensus.cc:513] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.644840 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae0d61d254c4e1ba508e30aa35600ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.645097 2553 leader_election.cc:290] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.645661 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b8a2d1a7f7146f4968590914374ba23" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:59.646049 712 leader_election.cc:343] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.650668 1156 ts_tablet_manager.cc:1918] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.650758 1156 log.cc:1199] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/06e31520d3ad4c3395a59049755a4725
I20250629 01:51:59.651124 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2fcd67db9bb141f8a0447eb7e7ca8f63" } new_table_name: "default.Table_5794ea9989b54ab9bd7ecca30428c971" modify_external_catalogs: true
I20250629 01:51:59.651131 1159 tablet_bootstrap.cc:492] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.651589 1159 ts_tablet_manager.cc:1397] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.000s
I20250629 01:51:59.655134 1159 raft_consensus.cc:357] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.655696 1159 raft_consensus.cc:383] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.655889 1156 ts_tablet_manager.cc:1939] T 06e31520d3ad4c3395a59049755a4725 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.655901 1159 raft_consensus.cc:738] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.656689 1156 tablet_replica.cc:331] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.656855 1156 raft_consensus.cc:2241] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.656963 1156 raft_consensus.cc:2270] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.656718 1159 consensus_queue.cc:260] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.657399 1156 ts_tablet_manager.cc:1905] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.658358 1159 ts_tablet_manager.cc:1428] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:51:59.659039 886 tablet_service.cc:1515] Processing DeleteTablet for tablet c0b9afb9976541b2bc7ab188f35a62ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.659631 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 52be184d0480495a894442d6f72a0f12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.659759 880 tablet_service.cc:1515] Processing DeleteTablet for tablet a6a012ba891949538800581436ef5380 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.659852 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 648b0a0f742b4e7da9eaff6bd8795fc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.660173 886 tablet_service.cc:1515] Processing DeleteTablet for tablet d6bf3c7d0bb548e0b77ba8b569a72d0d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.662488 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b17172ab8b4e0dad36ebaac776dc03 with delete_type TABLET_DATA_DELETED (replaced by tablet 35a62c3fe4fb4220838deece8e46d074) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.663058 1043 raft_consensus.cc:2466] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:59.663259 1158 raft_consensus.cc:2241] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.663434 710 leader_election.cc:304] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.663511 1158 raft_consensus.cc:2270] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.663722 2553 raft_consensus.cc:2802] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.663797 2553 raft_consensus.cc:695] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:59.663842 1159 tablet_bootstrap.cc:492] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.663962 1158 ts_tablet_manager.cc:1905] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.663956 2553 consensus_queue.cc:237] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.609609 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 12d63924c5ff4a99ac62f89a9d29423b with delete_type TABLET_DATA_DELETED (replaced by tablet f4cdefd680414cbda42e4090e78b1fd2) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.611294 1155 tablet_bootstrap.cc:492] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.667686 1155 ts_tablet_manager.cc:1397] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20250629 01:51:59.667948 1159 tablet_bootstrap.cc:654] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.668520 1155 raft_consensus.cc:357] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.668694 1155 raft_consensus.cc:383] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.668745 1155 raft_consensus.cc:738] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.668936 1155 consensus_queue.cc:260] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.669351 1155 ts_tablet_manager.cc:1428] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:59.669552 1155 tablet_bootstrap.cc:492] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.670132 1158 ts_tablet_manager.cc:1918] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.670207 1158 log.cc:1199] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4d9ef77181ba48f3a6de16b945c8df1a
I20250629 01:51:59.670444 1158 ts_tablet_manager.cc:1939] T 4d9ef77181ba48f3a6de16b945c8df1a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:59.670679 1155 tablet_bootstrap.cc:654] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.670660 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d195c53fd4e9475399af824a4a19d9de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.671041 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d195c53fd4e9475399af824a4a19d9de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.671036 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1215) took 1181 ms. Trace:
I20250629 01:51:59.671109 1158 rpcz_store.cc:276] 0629 01:51:58.489473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.489506 (+ 33us) service_pool.cc:225] Handling call
0629 01:51:59.481457 (+991951us) ts_tablet_manager.cc:1145] Deleting tablet 4d9ef77181ba48f3a6de16b945c8df1a
0629 01:51:59.481467 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:59.663425 (+181958us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x616000241054. stack: 00007f406fe3f316 00007f406fe3f0d0 00007f406ee353ad 0000000000971004 0000000000970fcd 000000000097099a 00007f407b8a0320 00007f407cbb2093 00007f407f68f137 00007f407f6c11c4 00007f407f6a2c20 00007f407f6a29ad 0000000000a7573c 00007f406fe849bf 00007f406fe87c8b 00007f406fe87b08
0629 01:51:59.666688 (+ 3263us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.670071 (+ 3383us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.671019 (+ 948us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":417245952,"tablet-delete.queue_time_us":991799}
I20250629 01:51:59.672140 2454 raft_consensus.cc:491] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.672256 2454 raft_consensus.cc:513] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.672765 2454 leader_election.cc:290] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:51:59.673585 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b8a2d1a7f7146f4968590914374ba23" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:51:59.673873 850 leader_election.cc:343] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 648b0a0f742b4e7da9eaff6bd8795fc4
I20250629 01:51:59.673982 1051 raft_consensus.cc:2466] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:51:59.674484 850 leader_election.cc:343] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 648b0a0f742b4e7da9eaff6bd8795fc4
I20250629 01:51:59.674573 850 leader_election.cc:304] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.674806 2454 raft_consensus.cc:2747] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.676019 710 leader_election.cc:304] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.676589 2534 raft_consensus.cc:2802] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.676657 2534 raft_consensus.cc:491] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.676714 2534 raft_consensus.cc:3058] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.678843 2534 raft_consensus.cc:513] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.679699 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b8a2d1a7f7146f4968590914374ba23" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:59.680159 712 leader_election.cc:343] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.680672 2534 leader_election.cc:290] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.681226 1043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b8a2d1a7f7146f4968590914374ba23" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
W20250629 01:51:59.681247 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 61108.4us
I20250629 01:51:59.681360 1043 raft_consensus.cc:3058] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.682260 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4d9ef77181ba48f3a6de16b945c8df1a (table default.Table_f2175d9af5064903ab1c97c2035e2527 [id=207fae1412fa4f1da7eb67dadd6f986e]) successfully deleted
I20250629 01:51:59.682686 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bb76c63ce749078365420f224d62f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.683738 1043 raft_consensus.cc:2466] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:51:59.684041 710 leader_election.cc:304] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.684520 2548 raft_consensus.cc:2802] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.684667 2548 raft_consensus.cc:695] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:51:59.684919 2548 consensus_queue.cc:237] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:59.686218 988 consensus_peers.cc:489] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:59.686967 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 46bb76c63ce749078365420f224d62f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.688014 1159 tablet_bootstrap.cc:492] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.688079 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 24e9caef62f64a4281789aba8f83386a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.688244 1159 ts_tablet_manager.cc:1397] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20250629 01:51:59.689028 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 24e9caef62f64a4281789aba8f83386a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.689254 1159 raft_consensus.cc:357] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.689477 1159 raft_consensus.cc:383] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.689592 1159 raft_consensus.cc:738] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.689765 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet c4c4153b6dcd45ada716c05d0590d10a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.689860 1159 consensus_queue.cc:260] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.690588 750 tablet_service.cc:1515] Processing DeleteTablet for tablet c4c4153b6dcd45ada716c05d0590d10a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.690593 1159 ts_tablet_manager.cc:1428] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:51:59.690635 2578 tablet.cc:1722] T 95247cc778504f4d85d9013f84c59e69 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.690872 1159 tablet_bootstrap.cc:492] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.691385 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae with delete_type TABLET_DATA_DELETED (replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.691737 1158 tablet_replica.cc:331] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:59.691901 1158 raft_consensus.cc:2241] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:59.692191 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.692885 750 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b94f8250a143b096703f2b0386e3cd with delete_type TABLET_DATA_DELETED (replaced by tablet c5771fdd618e45118df2b6dab7f668d9) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.693696 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 40592f11f430469f9c8e82eba25b38ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.694520 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.694638 1158 pending_rounds.cc:70] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:51:59.694834 1158 pending_rounds.cc:77] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759216852070400 op_type: NO_OP noop_request { }
I20250629 01:51:59.695045 1158 raft_consensus.cc:2887] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:59.695242 1158 raft_consensus.cc:2270] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.696715 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.696947 1159 tablet_bootstrap.cc:654] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.697333 750 tablet_service.cc:1515] Processing DeleteTablet for tablet f3bdc8bda8a940be865679a05b90bd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.697427 1158 ts_tablet_manager.cc:1905] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.698040 750 tablet_service.cc:1515] Processing DeleteTablet for tablet daf312e60c014fe698d3ec9f1a1b769c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.698601 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 819ed44d98f14075b9a67b88dc44e86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.703909 2578 tablet.cc:1722] T 5860f7100a78476b8fa1d34f6c0b8d77 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:59.713492 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 504c15b3b168499d9a82584f82a0b792 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c34a9c407304a6e825c61d81bde6ac2) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.714004 1159 tablet_bootstrap.cc:492] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.714210 1159 ts_tablet_manager.cc:1397] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250629 01:51:59.714586 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 504c15b3b168499d9a82584f82a0b792 with delete_type TABLET_DATA_DELETED (replaced by tablet 4c34a9c407304a6e825c61d81bde6ac2) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.715425 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae0d61d254c4e1ba508e30aa35600ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.716966 1155 tablet_bootstrap.cc:492] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.717070 1155 ts_tablet_manager.cc:1397] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.001s
I20250629 01:51:59.718114 1155 raft_consensus.cc:357] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.718254 1155 raft_consensus.cc:383] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.718303 1155 raft_consensus.cc:738] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.718453 1155 consensus_queue.cc:260] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:51:59.719559 2526 ts_tablet_manager.cc:1918] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.722079 2526 log.cc:1199] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/da93c730451d42bc8df1e582fb3af818
I20250629 01:51:59.717796 2527 consensus_queue.cc:1035] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250629 01:51:59.722534 2526 ts_tablet_manager.cc:1939] T da93c730451d42bc8df1e582fb3af818 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:59.723233 2526 tablet_replica.cc:331] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.723322 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet da93c730451d42bc8df1e582fb3af818 (table default.Table_2e7ef5453b1945728384798f25e47b7f [id=88472d60f4244733beb9ff0fd6ce9c9f]) successfully deleted
I20250629 01:51:59.723851 2526 raft_consensus.cc:2241] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.724193 2526 raft_consensus.cc:2270] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.724988 2526 ts_tablet_manager.cc:1905] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.724007 1155 ts_tablet_manager.cc:1428] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:59.728440 1155 tablet_bootstrap.cc:492] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.725718 1159 raft_consensus.cc:357] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.728789 1159 raft_consensus.cc:383] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.728946 1159 raft_consensus.cc:738] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.729209 1159 consensus_queue.cc:260] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.729853 1155 tablet_bootstrap.cc:654] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.729878 1159 ts_tablet_manager.cc:1428] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250629 01:51:59.730530 1159 tablet_bootstrap.cc:492] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.734687 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae0d61d254c4e1ba508e30aa35600ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.735292 2454 raft_consensus.cc:491] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.735399 2454 raft_consensus.cc:513] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.735827 2454 leader_election.cc:290] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:51:59.736521 850 leader_election.cc:343] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0ba9eabd06174b87a1485f4921895c5f
I20250629 01:51:59.737164 1159 tablet_bootstrap.cc:654] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:59.738634 850 leader_election.cc:343] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0ba9eabd06174b87a1485f4921895c5f
I20250629 01:51:59.738718 850 leader_election.cc:304] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.739038 2454 raft_consensus.cc:2747] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.747429 2526 ts_tablet_manager.cc:1918] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.749879 2526 log.cc:1199] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/2aca26af31ed4bc994f4d62603edc881
I20250629 01:51:59.750255 2526 ts_tablet_manager.cc:1939] T 2aca26af31ed4bc994f4d62603edc881 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:59.750438 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 6092e19acf974f2dad5a903da72944ba (table default.Table_913feb22c480457e96599a6814b30c60 [id=abbf099fb0aa4b118f76e5c48700df39]) successfully deleted
I20250629 01:51:59.751012 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 06e31520d3ad4c3395a59049755a4725 (table default.Table_2e69d514f50649f79824a1cb1da60444 [id=d843c94cf5f847249d82ea7944980d89]) successfully deleted
I20250629 01:51:59.748940 1159 tablet_bootstrap.cc:492] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.751444 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 2aca26af31ed4bc994f4d62603edc881 (table default.Table_8bf46b174da647d2b1135f050ce5e430 [id=72ebf1394e9a40faaee64e1b26e6ffe6]) successfully deleted
W20250629 01:51:59.751686 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 504c15b3b168499d9a82584f82a0b792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 504c15b3b168499d9a82584f82a0b792 already in progress: creating tablet
I20250629 01:51:59.751888 1156 ts_tablet_manager.cc:1918] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:51:59.751893 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8ae0d61d254c4e1ba508e30aa35600ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae0d61d254c4e1ba508e30aa35600ca already in progress: creating tablet
I20250629 01:51:59.751081 2526 tablet_replica.cc:331] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.752621 2001 ts_manager.cc:194] Registered new tserver with Master: 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:59.753820 1156 log.cc:1199] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/258994dd24804f4aaa5e299436fac9b0
I20250629 01:51:59.751247 1159 ts_tablet_manager.cc:1397] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20250629 01:51:59.754316 1156 ts_tablet_manager.cc:1939] T 258994dd24804f4aaa5e299436fac9b0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.754503 1159 raft_consensus.cc:357] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.754678 1159 raft_consensus.cc:383] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.754761 1159 raft_consensus.cc:738] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.754981 1159 consensus_queue.cc:260] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.755436 1159 ts_tablet_manager.cc:1428] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:51:59.758785 1159 tablet_bootstrap.cc:492] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.755609 2526 raft_consensus.cc:2241] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.759266 2526 raft_consensus.cc:2270] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.760408 2526 ts_tablet_manager.cc:1905] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.760906 1154 tablet_bootstrap.cc:492] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:59.761044 1154 ts_tablet_manager.cc:1397] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.442s user 0.010s sys 0.009s
W20250629 01:51:59.761548 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b847d6ea2594d1880f4d7a9d7e825e8 consensus state has no leader
I20250629 01:51:59.761684 1154 raft_consensus.cc:357] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.761849 1154 raft_consensus.cc:383] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.761925 1154 raft_consensus.cc:738] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.763513 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4658a2e500d648bc9e25fd35a86d523f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.763949 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4658a2e500d648bc9e25fd35a86d523f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.764470 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ed0a310f4015427690bfcc25f7618ada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.764863 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed0a310f4015427690bfcc25f7618ada in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.764832 1154 consensus_queue.cc:260] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.765010 1159 tablet_bootstrap.cc:654] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.765323 1154 ts_tablet_manager.cc:1428] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250629 01:51:59.765578 1154 tablet_bootstrap.cc:492] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:51:59.765857 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50f2d76bc0a041c79b058c537150b894 consensus state has no leader
W20250629 01:51:59.766371 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30265f664bff4b058e587edd9e0c5978 consensus state has no leader
W20250629 01:51:59.766646 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76118a6e684448b09f244d44c909ccaa consensus state has no leader
I20250629 01:51:59.766880 1154 tablet_bootstrap.cc:654] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.767951 1155 tablet_bootstrap.cc:492] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.768054 1155 ts_tablet_manager.cc:1397] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.001s
I20250629 01:51:59.768674 1155 raft_consensus.cc:357] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.771553 2487 tablet.cc:1722] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.774134 2526 ts_tablet_manager.cc:1918] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.774209 2526 log.cc:1199] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/c0b9afb9976541b2bc7ab188f35a62ec
I20250629 01:51:59.774456 2526 ts_tablet_manager.cc:1939] T c0b9afb9976541b2bc7ab188f35a62ec P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:59.776417 2578 tablet.cc:1722] T 8bb4d7c2096a45df8eb392a2e7d85c3d P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.776953 2526 tablet_replica.cc:331] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.775872 2558 tablet.cc:1722] T 8bb4d7c2096a45df8eb392a2e7d85c3d P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:51:59.777055 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet c0b9afb9976541b2bc7ab188f35a62ec (table default.Table_d6eb1cb53be64d2d81c86f31b21bbecc [id=7edb7d438b5b4ab3b529cb31fc761610]) successfully deleted
I20250629 01:51:59.776903 1156 tablet_replica.cc:331] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.777544 1156 raft_consensus.cc:2241] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.777694 1156 raft_consensus.cc:2270] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.778216 1156 ts_tablet_manager.cc:1905] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.780951 1155 raft_consensus.cc:383] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.781047 1155 raft_consensus.cc:738] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.781279 2526 raft_consensus.cc:2241] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.781263 1155 consensus_queue.cc:260] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.781697 1155 ts_tablet_manager.cc:1428] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250629 01:51:59.781939 1155 tablet_bootstrap.cc:492] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.781975 2526 raft_consensus.cc:2270] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.782757 2526 ts_tablet_manager.cc:1905] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.783223 1155 tablet_bootstrap.cc:654] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.786315 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_63d441b893a5478e83e7691f2e9dc343"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.786725 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63d441b893a5478e83e7691f2e9dc343 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.794422 1156 ts_tablet_manager.cc:1918] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.794886 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1fcaf9bb8e77453f820db3cd77028eaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.795286 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fcaf9bb8e77453f820db3cd77028eaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.796180 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6cbce56fa7a54b0eb7109265617d47eb" } modify_external_catalogs: true
I20250629 01:51:59.796306 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6cbce56fa7a54b0eb7109265617d47eb" } modify_external_catalogs: true
I20250629 01:51:59.798257 1156 log.cc:1199] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/3a81e6f9711c4d398ae09f33a75566ae
I20250629 01:51:59.798552 1156 ts_tablet_manager.cc:1939] T 3a81e6f9711c4d398ae09f33a75566ae P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.795300 1155 tablet_bootstrap.cc:492] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.800904 1155 ts_tablet_manager.cc:1397] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250629 01:51:59.801509 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 258994dd24804f4aaa5e299436fac9b0 (table default.Table_a8bede9e061941eaabacba34109b16ab [id=c4001552f202478e92afae5c3aacfa9b]) successfully deleted
I20250629 01:51:59.801558 1155 raft_consensus.cc:357] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:59.801714 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7aef2514a87c4c4aaf4d51b314630185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7aef2514a87c4c4aaf4d51b314630185 already in progress: creating tablet
I20250629 01:51:59.801815 1155 raft_consensus.cc:383] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:51:59.801883 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ab59d0a03543499fbacaf39743ca812b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab59d0a03543499fbacaf39743ca812b already in progress: creating tablet
I20250629 01:51:59.801903 1155 raft_consensus.cc:738] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
W20250629 01:51:59.802038 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f7e6f33c7bee44d1a2b6feb0625faedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7e6f33c7bee44d1a2b6feb0625faedf already in progress: creating tablet
W20250629 01:51:59.802186 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 12d63924c5ff4a99ac62f89a9d29423b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12d63924c5ff4a99ac62f89a9d29423b already in progress: creating tablet
I20250629 01:51:59.802206 1155 consensus_queue.cc:260] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:59.802340 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 46bb76c63ce749078365420f224d62f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46bb76c63ce749078365420f224d62f3 already in progress: creating tablet
I20250629 01:51:59.802803 1155 ts_tablet_manager.cc:1428] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:51:59.803092 1155 tablet_bootstrap.cc:492] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.803138 2580 raft_consensus.cc:491] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.803239 2580 raft_consensus.cc:513] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.808982 2580 leader_election.cc:290] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.809522 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8975c1deaaf44761be791f460b53ec74" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:59.809671 775 raft_consensus.cc:2466] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:59.810050 986 leader_election.cc:304] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:59.810253 2580 raft_consensus.cc:2802] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.810321 2580 raft_consensus.cc:491] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.810379 2580 raft_consensus.cc:3058] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.812572 1156 tablet_replica.cc:331] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.812554 2580 raft_consensus.cc:513] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.812810 1156 raft_consensus.cc:2241] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:59.813019 1156 pending_rounds.cc:70] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:59.813051 2580 leader_election.cc:290] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.813158 1156 pending_rounds.cc:77] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759222945087488 op_type: NO_OP noop_request { }
I20250629 01:51:59.813333 1156 raft_consensus.cc:2887] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:59.813453 1156 raft_consensus.cc:2270] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.813583 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8975c1deaaf44761be791f460b53ec74" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:59.813702 775 raft_consensus.cc:3058] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.814325 1156 ts_tablet_manager.cc:1905] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.815855 775 raft_consensus.cc:2466] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:59.816427 986 leader_election.cc:304] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:51:59.816530 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 24e9caef62f64a4281789aba8f83386a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24e9caef62f64a4281789aba8f83386a already in progress: creating tablet
I20250629 01:51:59.816701 1155 tablet_bootstrap.cc:654] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:51:59.816704 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c4c4153b6dcd45ada716c05d0590d10a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4c4153b6dcd45ada716c05d0590d10a already in progress: creating tablet
I20250629 01:51:59.816828 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 3a81e6f9711c4d398ae09f33a75566ae (table default.Table_67f686028f0c40c991d3231bd0b0c823 [id=c51c68dd349543d4a4a8bfb1b9aa1184]) successfully deleted
I20250629 01:51:59.821120 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:51:59.821647 452 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet cd9a41fae477470281dfba4d4e552b99 (table default.Table_c58b64be9b9040a294ce611a1bd282c7 [id=4525666163d64ba981dfff1c095c6ee2]). Expected version 1 got 0
I20250629 01:51:59.823009 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:51:59.823128 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d (table default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a]). Expected version 2 got 0
I20250629 01:51:59.824259 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_4f1e7e5479c84d75868d25a5c2745c88 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 1 got 0
I20250629 01:51:59.824405 446 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 95247cc778504f4d85d9013f84c59e69 (table default.Table_20ba093e340e4fe7ae570398e255a99b [id=eeb89fb29ca74bd89184ad0a5e2d8c8b]). Expected version 2 got 0
I20250629 01:51:59.777870 1158 ts_tablet_manager.cc:1918] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:59.825582 1158 log.cc:1199] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/68cedbd093b4420d8f7b29219ce8f77b
I20250629 01:51:59.826092 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7fff96135543443f861dc133aea7b971" } modify_external_catalogs: true
I20250629 01:51:59.826215 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7fff96135543443f861dc133aea7b971" } modify_external_catalogs: true
I20250629 01:51:59.826669 1158 ts_tablet_manager.cc:1939] T 68cedbd093b4420d8f7b29219ce8f77b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:59.827291 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1216) took 1337 ms. Trace:
I20250629 01:51:59.828382 750 tablet_service.cc:1468] Processing CreateTablet for tablet e36bd721dbcf4f99a361678edf0d3a64 (DEFAULT_TABLE table=default.Table_5e8950b2ec654796865087692160d831 [id=6e3e0a40d5d540c0b3bfcd6705931598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.828677 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36bd721dbcf4f99a361678edf0d3a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.827697 1017 tablet_service.cc:1468] Processing CreateTablet for tablet e36bd721dbcf4f99a361678edf0d3a64 (DEFAULT_TABLE table=default.Table_5e8950b2ec654796865087692160d831 [id=6e3e0a40d5d540c0b3bfcd6705931598]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.833050 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36bd721dbcf4f99a361678edf0d3a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.833408 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aca8c2c6ca5045838333bfd1e7d46d50" } modify_external_catalogs: true
I20250629 01:51:59.833515 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aca8c2c6ca5045838333bfd1e7d46d50" } modify_external_catalogs: true
I20250629 01:51:59.834352 1156 ts_tablet_manager.cc:1918] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:59.834432 1156 log.cc:1199] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4d9ef77181ba48f3a6de16b945c8df1a
I20250629 01:51:59.834388 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d195c53fd4e9475399af824a4a19d9de" } modify_external_catalogs: true
I20250629 01:51:59.834492 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d195c53fd4e9475399af824a4a19d9de" } modify_external_catalogs: true
I20250629 01:51:59.834700 1156 ts_tablet_manager.cc:1939] T 4d9ef77181ba48f3a6de16b945c8df1a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.835040 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6794618e8690495596d2c46dc14125b2" } new_table_name: "default.Table_3f47c00ffce0403692345454e7fa9225" modify_external_catalogs: true
I20250629 01:51:59.835295 1155 tablet_bootstrap.cc:492] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.835387 1155 ts_tablet_manager.cc:1397] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20250629 01:51:59.835952 1155 raft_consensus.cc:357] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.836074 1155 raft_consensus.cc:383] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.836166 1155 raft_consensus.cc:738] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.836344 1155 consensus_queue.cc:260] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.836817 1155 ts_tablet_manager.cc:1428] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250629 01:51:59.837034 1155 tablet_bootstrap.cc:492] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.839044 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 12efda19b5ee445f968a5671522295f5 (DEFAULT_TABLE table=default.Table_d1dc5307aa7c47738a408a2ebd00a685 [id=59b41c8d3e09451c93f70add0df7cd33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.839247 1158 rpcz_store.cc:276] 0629 01:51:58.489802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.489842 (+ 40us) service_pool.cc:225] Handling call
0629 01:51:59.691696 (+1201854us) ts_tablet_manager.cc:1145] Deleting tablet 68cedbd093b4420d8f7b29219ce8f77b
0629 01:51:59.691707 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:59.727887 (+ 36180us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.777800 (+ 49913us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.827275 (+ 49475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1201731}
I20250629 01:51:59.840384 750 tablet_service.cc:1468] Processing CreateTablet for tablet 12efda19b5ee445f968a5671522295f5 (DEFAULT_TABLE table=default.Table_d1dc5307aa7c47738a408a2ebd00a685 [id=59b41c8d3e09451c93f70add0df7cd33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.840503 1155 tablet_bootstrap.cc:654] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.839810 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12efda19b5ee445f968a5671522295f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.840674 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12efda19b5ee445f968a5671522295f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.844234 1156 tablet_replica.cc:331] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.844415 1156 raft_consensus.cc:2241] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.844563 1156 raft_consensus.cc:2270] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.844955 1156 ts_tablet_manager.cc:1905] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.847803 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 16a4ba7e670a4445b6304f4d6d95b0e6 (DEFAULT_TABLE table=default.Table_ec52356e4f4045829090d362590a9ca2 [id=7fa4f8aacbdb418186166921799d36f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.848137 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a4ba7e670a4445b6304f4d6d95b0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.848053 752 tablet_service.cc:1468] Processing CreateTablet for tablet 16a4ba7e670a4445b6304f4d6d95b0e6 (DEFAULT_TABLE table=default.Table_ec52356e4f4045829090d362590a9ca2 [id=7fa4f8aacbdb418186166921799d36f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.848371 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a4ba7e670a4445b6304f4d6d95b0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.854122 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 412a3d30c85442cd943a163cf5ea61b5 (DEFAULT_TABLE table=default.Table_702cf8b71dad4ba9a893b84bf903252b [id=102bfa02489d46738e10d4916786d347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.854429 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412a3d30c85442cd943a163cf5ea61b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.854451 752 tablet_service.cc:1468] Processing CreateTablet for tablet 412a3d30c85442cd943a163cf5ea61b5 (DEFAULT_TABLE table=default.Table_702cf8b71dad4ba9a893b84bf903252b [id=102bfa02489d46738e10d4916786d347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.854766 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412a3d30c85442cd943a163cf5ea61b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.859779 1154 tablet_bootstrap.cc:492] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:59.859887 1154 ts_tablet_manager.cc:1397] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.001s
I20250629 01:51:59.859889 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 1073fbe267ad48d48bfe17f29dabe3b3 (DEFAULT_TABLE table=default.Table_fd765b36942a425a89bfdd90ae5efd35 [id=c43f8ce16a0f4c59b5e92b870017d61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.860234 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1073fbe267ad48d48bfe17f29dabe3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.860383 752 tablet_service.cc:1468] Processing CreateTablet for tablet 1073fbe267ad48d48bfe17f29dabe3b3 (DEFAULT_TABLE table=default.Table_fd765b36942a425a89bfdd90ae5efd35 [id=c43f8ce16a0f4c59b5e92b870017d61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.860632 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1073fbe267ad48d48bfe17f29dabe3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.860605 1154 raft_consensus.cc:357] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.860718 1154 raft_consensus.cc:383] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.860752 1154 raft_consensus.cc:738] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.860895 1154 consensus_queue.cc:260] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.861263 1154 ts_tablet_manager.cc:1428] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:51:59.861465 1154 tablet_bootstrap.cc:492] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:59.861909 2580 raft_consensus.cc:2802] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.862021 2580 raft_consensus.cc:695] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:59.862214 2580 consensus_queue.cc:237] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.862695 1154 tablet_bootstrap.cc:654] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.863112 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 475ea15bc975451a83e5543ae94d6dfe (DEFAULT_TABLE table=default.Table_af895bd9f36e4ed4b365f6d299c7f249 [id=aa0961c80d084cd498a003384ff41444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.863361 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475ea15bc975451a83e5543ae94d6dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.864210 2565 raft_consensus.cc:491] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.864305 2565 raft_consensus.cc:513] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.864876 2565 leader_election.cc:290] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.865212 2565 raft_consensus.cc:491] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.865301 2565 raft_consensus.cc:513] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.865716 2565 leader_election.cc:290] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.866340 742 tablet_service.cc:1468] Processing CreateTablet for tablet 475ea15bc975451a83e5543ae94d6dfe (DEFAULT_TABLE table=default.Table_af895bd9f36e4ed4b365f6d299c7f249 [id=aa0961c80d084cd498a003384ff41444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.866608 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475ea15bc975451a83e5543ae94d6dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.867954 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4d9ef77181ba48f3a6de16b945c8df1a (table default.Table_f2175d9af5064903ab1c97c2035e2527 [id=207fae1412fa4f1da7eb67dadd6f986e]) successfully deleted
I20250629 01:51:59.871856 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe9674ffeb44ba6ab60de59d4c08670" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:59.871992 774 raft_consensus.cc:2466] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:59.872188 742 tablet_service.cc:1468] Processing CreateTablet for tablet 462bd96ccd1e41dd8ff3fb389c7f7a18 (DEFAULT_TABLE table=default.Table_e3c425798b1f45acaa203c1a4dcad1c1 [id=3d663a94f945429891c9319250f7ac15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.872359 986 leader_election.cc:304] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:59.872563 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462bd96ccd1e41dd8ff3fb389c7f7a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.875160 1156 ts_tablet_manager.cc:1918] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.875238 1156 log.cc:1199] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e3b94f8250a143b096703f2b0386e3cd
I20250629 01:51:59.875454 1156 ts_tablet_manager.cc:1939] T e3b94f8250a143b096703f2b0386e3cd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.876554 2565 raft_consensus.cc:2802] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.876631 2565 raft_consensus.cc:491] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.876694 2565 raft_consensus.cc:3058] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.877108 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd9a41fae477470281dfba4d4e552b99" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:59.877266 775 raft_consensus.cc:2466] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:59.877696 986 leader_election.cc:304] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:51:59.877928 2527 raft_consensus.cc:2802] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.877987 2527 raft_consensus.cc:491] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.878046 2527 raft_consensus.cc:3058] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.878782 742 tablet_service.cc:1468] Processing CreateTablet for tablet 2b3045aa142e4471a79de74a5a272e36 (DEFAULT_TABLE table=default.Table_2d2750afeab44fda88c390b72b529bcc [id=815390b29fd641ed9a862921ae94fcfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.879104 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3045aa142e4471a79de74a5a272e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.882172 2565 raft_consensus.cc:513] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.882473 2527 raft_consensus.cc:513] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.883049 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2b3045aa142e4471a79de74a5a272e36 (DEFAULT_TABLE table=default.Table_2d2750afeab44fda88c390b72b529bcc [id=815390b29fd641ed9a862921ae94fcfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.883391 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 462bd96ccd1e41dd8ff3fb389c7f7a18 (DEFAULT_TABLE table=default.Table_e3c425798b1f45acaa203c1a4dcad1c1 [id=3d663a94f945429891c9319250f7ac15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.883631 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462bd96ccd1e41dd8ff3fb389c7f7a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.809376 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8975c1deaaf44761be791f460b53ec74" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:59.813407 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8975c1deaaf44761be791f460b53ec74" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.824774 886 tablet_service.cc:1468] Processing CreateTablet for tablet e36bd721dbcf4f99a361678edf0d3a64 (DEFAULT_TABLE table=default.Table_5e8950b2ec654796865087692160d831 [id=6e3e0a40d5d540c0b3bfcd6705931598]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:51:59.885250 988 leader_election.cc:343] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.885448 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36bd721dbcf4f99a361678edf0d3a64. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:51:59.885582 988 leader_election.cc:343] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.824982 890 tablet_service.cc:1468] Processing CreateTablet for tablet 475ea15bc975451a83e5543ae94d6dfe (DEFAULT_TABLE table=default.Table_af895bd9f36e4ed4b365f6d299c7f249 [id=aa0961c80d084cd498a003384ff41444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.888386 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 475ea15bc975451a83e5543ae94d6dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.890014 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e3b94f8250a143b096703f2b0386e3cd (table default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]) successfully deleted
I20250629 01:51:59.890479 1156 tablet_replica.cc:331] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.890650 1156 raft_consensus.cc:2241] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:51:59.890856 1156 pending_rounds.cc:70] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:51:59.890897 1156 pending_rounds.cc:77] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759221572210688 op_type: NO_OP noop_request { }
I20250629 01:51:59.890974 1156 raft_consensus.cc:2887] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:51:59.891045 1156 raft_consensus.cc:2270] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.824982 880 tablet_service.cc:1468] Processing CreateTablet for tablet 12efda19b5ee445f968a5671522295f5 (DEFAULT_TABLE table=default.Table_d1dc5307aa7c47738a408a2ebd00a685 [id=59b41c8d3e09451c93f70add0df7cd33]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.892340 1158 tablet_replica.cc:331] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:51:59.892437 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12efda19b5ee445f968a5671522295f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.892505 1158 raft_consensus.cc:2241] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.892608 1158 raft_consensus.cc:2270] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.892993 1158 ts_tablet_manager.cc:1905] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.893570 1156 ts_tablet_manager.cc:1905] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.893635 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0" } modify_external_catalogs: true
I20250629 01:51:59.893751 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0" } modify_external_catalogs: true
I20250629 01:51:59.895023 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_cbb81a42b45040139dcda6493db2fe5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.895444 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cbb81a42b45040139dcda6493db2fe5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.825129 884 tablet_service.cc:1468] Processing CreateTablet for tablet 16a4ba7e670a4445b6304f4d6d95b0e6 (DEFAULT_TABLE table=default.Table_ec52356e4f4045829090d362590a9ca2 [id=7fa4f8aacbdb418186166921799d36f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.896333 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a4ba7e670a4445b6304f4d6d95b0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.897759 2578 tablet.cc:1722] T 013ab393a0f14bdc89389349bcc5d24b P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:59.899245 2558 tablet.cc:1722] T 013ab393a0f14bdc89389349bcc5d24b P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:59.825414 885 tablet_service.cc:1468] Processing CreateTablet for tablet 412a3d30c85442cd943a163cf5ea61b5 (DEFAULT_TABLE table=default.Table_702cf8b71dad4ba9a893b84bf903252b [id=102bfa02489d46738e10d4916786d347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.825924 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1073fbe267ad48d48bfe17f29dabe3b3 (DEFAULT_TABLE table=default.Table_fd765b36942a425a89bfdd90ae5efd35 [id=c43f8ce16a0f4c59b5e92b870017d61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.829561 896 tablet_service.cc:1468] Processing CreateTablet for tablet 462bd96ccd1e41dd8ff3fb389c7f7a18 (DEFAULT_TABLE table=default.Table_e3c425798b1f45acaa203c1a4dcad1c1 [id=3d663a94f945429891c9319250f7ac15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.830103 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2b3045aa142e4471a79de74a5a272e36 (DEFAULT_TABLE table=default.Table_2d2750afeab44fda88c390b72b529bcc [id=815390b29fd641ed9a862921ae94fcfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:51:59.871518 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd9a41fae477470281dfba4d4e552b99" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:59.871687 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe9674ffeb44ba6ab60de59d4c08670" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:51:59.901350 988 leader_election.cc:343] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:59.901623 988 leader_election.cc:343] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.907859 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 68cedbd093b4420d8f7b29219ce8f77b (table default.Table_31aef3786c3841c096a18ca7abfa40df [id=b46cf5dd7d524d0cb23f236b318eafca]) successfully deleted
I20250629 01:51:59.909863 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412a3d30c85442cd943a163cf5ea61b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.912215 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 462bd96ccd1e41dd8ff3fb389c7f7a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.912799 1155 tablet_bootstrap.cc:492] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:51:59.912899 1155 ts_tablet_manager.cc:1397] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.007s
I20250629 01:51:59.913252 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 77f6bab8deb740cc9cf43832196e404c
I20250629 01:51:59.913503 1155 raft_consensus.cc:357] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.913667 1155 raft_consensus.cc:383] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.913722 1155 raft_consensus.cc:738] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.913828 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1d228257fd614e23b098ddd7c3e7d0d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:59.913892 1155 consensus_queue.cc:260] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:51:59.914242 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d228257fd614e23b098ddd7c3e7d0d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.915076 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929] alter complete (version 2)
I20250629 01:51:59.915208 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_f705303395b14b0daf27cb67a19275a8 [id=606dee2396e645cd829e21ff5b18e546] alter complete (version 1)
I20250629 01:51:59.916384 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1073fbe267ad48d48bfe17f29dabe3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.916071 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c969377aaf034e9b896c1b53366555ff
W20250629 01:51:59.916355 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12873ec9a5574e789f9d669ddda85704 consensus state has no leader
I20250629 01:51:59.919924 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bbe1f8083a884cdd97afce83bd22073c
I20250629 01:51:59.920044 1155 ts_tablet_manager.cc:1428] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:51:59.920279 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3045aa142e4471a79de74a5a272e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.920753 454 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet d093195a99244ee3a1f2c1678c95a9e0
I20250629 01:51:59.922037 1155 tablet_bootstrap.cc:492] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:51:59.922904 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3045aa142e4471a79de74a5a272e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:51:59.923055 2565 leader_election.cc:290] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.923126 2527 leader_election.cc:290] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.923566 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd9a41fae477470281dfba4d4e552b99" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.924199 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe9674ffeb44ba6ab60de59d4c08670" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:51:59.924799 988 leader_election.cc:343] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:51:59.924970 988 leader_election.cc:343] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.925765 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ba9eabd06174b87a1485f4921895c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.927925 878 tablet_service.cc:1515] Processing DeleteTablet for tablet c969377aaf034e9b896c1b53366555ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.927987 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd9a41fae477470281dfba4d4e552b99" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:59.928078 883 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe1f8083a884cdd97afce83bd22073c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.928314 775 raft_consensus.cc:3058] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.929194 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fe9674ffeb44ba6ab60de59d4c08670" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:59.929316 777 raft_consensus.cc:3058] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.930488 775 raft_consensus.cc:2466] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:59.930965 986 leader_election.cc:304] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.931309 2527 raft_consensus.cc:2802] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.931407 2527 raft_consensus.cc:695] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:59.931610 777 raft_consensus.cc:2466] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:51:59.931587 2527 consensus_queue.cc:237] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.931905 986 leader_election.cc:304] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.932279 2567 raft_consensus.cc:2802] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.932456 2567 raft_consensus.cc:695] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:59.932752 2567 consensus_queue.cc:237] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.935020 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 77f6bab8deb740cc9cf43832196e404c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.935792 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 77f6bab8deb740cc9cf43832196e404c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.936559 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.937444 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c969377aaf034e9b896c1b53366555ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.938177 750 tablet_service.cc:1515] Processing DeleteTablet for tablet c969377aaf034e9b896c1b53366555ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.938602 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe1f8083a884cdd97afce83bd22073c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:51:59.938741 2458 raft_consensus.cc:491] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.938835 2458 raft_consensus.cc:513] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.939297 2458 leader_election.cc:290] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.939303 750 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe1f8083a884cdd97afce83bd22073c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.939729 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45ca747f03524076b4112e4db78614fe" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:51:59.940021 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c995ce14c9a4966b842fcba520cf87d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:59.940052 988 leader_election.cc:343] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.940280 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 77f6bab8deb740cc9cf43832196e404c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.939914 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45ca747f03524076b4112e4db78614fe" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:51:59.940598 777 raft_consensus.cc:2466] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:51:59.940644 2487 tablet.cc:1722] T 013ab393a0f14bdc89389349bcc5d24b P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:51:59.940927 986 leader_election.cc:304] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.941247 2458 raft_consensus.cc:2802] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:51:59.941318 2458 raft_consensus.cc:491] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:51:59.941371 2458 raft_consensus.cc:3058] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.943463 2458 raft_consensus.cc:513] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.943917 2458 leader_election.cc:290] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:51:59.944604 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45ca747f03524076b4112e4db78614fe" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:51:59.944414 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "45ca747f03524076b4112e4db78614fe" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:51:59.944743 775 raft_consensus.cc:3058] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:51:59.945528 1159 tablet_bootstrap.cc:492] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:51:59.945626 1159 ts_tablet_manager.cc:1397] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.187s user 0.003s sys 0.004s
I20250629 01:51:59.946244 1159 raft_consensus.cc:357] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.946362 1159 raft_consensus.cc:383] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.946400 1159 raft_consensus.cc:738] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.946825 775 raft_consensus.cc:2466] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
W20250629 01:51:59.947208 988 leader_election.cc:343] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:51:59.947443 986 leader_election.cc:304] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:51:59.947736 2565 raft_consensus.cc:2802] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:51:59.947832 2565 raft_consensus.cc:695] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:51:59.946580 1159 consensus_queue.cc:260] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.948002 2565 consensus_queue.cc:237] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.948458 1159 ts_tablet_manager.cc:1428] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:51:59.948668 1159 tablet_bootstrap.cc:492] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:51:59.950081 1155 tablet_bootstrap.cc:654] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.950309 1159 tablet_bootstrap.cc:654] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.950698 1154 tablet_bootstrap.cc:492] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:51:59.951195 1154 ts_tablet_manager.cc:1397] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.090s user 0.000s sys 0.008s
I20250629 01:51:59.952082 1154 raft_consensus.cc:357] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.953835 1154 raft_consensus.cc:383] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:51:59.955456 1154 raft_consensus.cc:738] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:51:59.955511 1156 ts_tablet_manager.cc:1918] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:51:59.955832 1154 consensus_queue.cc:260] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.956261 1154 ts_tablet_manager.cc:1428] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:51:59.960737 1156 log.cc:1199] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/40592f11f430469f9c8e82eba25b38ed
I20250629 01:51:59.961099 1156 ts_tablet_manager.cc:1939] T 40592f11f430469f9c8e82eba25b38ed P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:51:59.961921 1154 tablet_bootstrap.cc:492] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:51:59.967358 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9c1dc6d6d2b1458cb57875d2e25354bc
I20250629 01:51:59.968195 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_c23a34743c38460887967df74d4a4bfb [id=658b17cdd5f649119d2c3dd59b8c619a] alter complete (version 2)
I20250629 01:51:59.970069 2526 ts_tablet_manager.cc:1918] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.970160 2526 log.cc:1199] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/52be184d0480495a894442d6f72a0f12
I20250629 01:51:59.970700 1154 tablet_bootstrap.cc:654] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:51:59.971843 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4f1e7e5479c84d75868d25a5c2745c88" } new_table_name: "default.Table_3c4a230556474a8e831d7180634f0155" modify_external_catalogs: true
I20250629 01:51:59.972873 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2a4864f0d4da45b7b83ae578092fc4d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.973328 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a4864f0d4da45b7b83ae578092fc4d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.975055 2454 raft_consensus.cc:491] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:51:59.975149 2454 raft_consensus.cc:513] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:51:59.975528 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1dc6d6d2b1458cb57875d2e25354bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.975639 2454 leader_election.cc:290] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:51:59.976176 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 56b60431b72e4792aa1c32fdee40cdb9 with delete_type TABLET_DATA_DELETED (replaced by tablet d73c2eb0d69e467b94a250e1790d8142) from {username='slave'} at 127.0.0.1:56376
I20250629 01:51:59.976205 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_fb78bf13072345fea20a096dffdba5fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:59.976310 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4320f615464fedbcab25137952f684 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:51:59.976608 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb78bf13072345fea20a096dffdba5fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.977662 1156 tablet_replica.cc:331] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:51:59.977653 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fd06736c67aa446890bfcc17305ecf16" } new_table_name: "default.Table_c4f6e00dfa43444cad42a37f26963e19" modify_external_catalogs: true
I20250629 01:51:59.977816 1156 raft_consensus.cc:2241] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.977939 1156 raft_consensus.cc:2270] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:51:59.977119 850 leader_election.cc:343] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 0d4320f615464fedbcab25137952f684
W20250629 01:51:59.978329 850 leader_election.cc:343] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 0d4320f615464fedbcab25137952f684
I20250629 01:51:59.978348 1156 ts_tablet_manager.cc:1905] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.978401 850 leader_election.cc:304] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:51:59.978564 2454 raft_consensus.cc:2747] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:51:59.978848 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c] alter complete (version 2)
I20250629 01:51:59.980366 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_073d8d6803d541acb4e3549a8ca2dca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.980720 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_073d8d6803d541acb4e3549a8ca2dca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:51:59.981652 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8461ee9ea7fa468dba21bab26f97cd23" } new_table_name: "default.Table_928d8362e4dc4845a90ee6811595c567" modify_external_catalogs: true
I20250629 01:51:59.982488 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_07e9a307bb3344089bc369d3f1bf6c34" } new_table_name: "default.Table_67bf8a3296bd47d7bf3977d6d088689c" modify_external_catalogs: true
I20250629 01:51:59.983213 1158 ts_tablet_manager.cc:1918] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:51:59.983294 1158 log.cc:1199] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/40592f11f430469f9c8e82eba25b38ed
I20250629 01:51:59.983633 1158 ts_tablet_manager.cc:1939] T 40592f11f430469f9c8e82eba25b38ed P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:51:59.983573 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_26700fd18fbf4cddb81d599cf016884c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:51:59.983915 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26700fd18fbf4cddb81d599cf016884c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:51:59.986546 986 consensus_peers.cc:489] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 -> Peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Couldn't send request to peer ef5c5979bb9e475f983af7c3e9e0e019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a81e6f9711c4d398ae09f33a75566ae. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:51:59.988430 2526 ts_tablet_manager.cc:1939] T 52be184d0480495a894442d6f72a0f12 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:51:59.989018 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 52be184d0480495a894442d6f72a0f12 (table default.Table_cbbc85d10dc546b59c06dd6296eaa2db [id=fadb6eabbead48b1a11795ffed0c4d81]) successfully deleted
I20250629 01:51:59.989091 2526 tablet_replica.cc:331] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:51:59.989248 2526 raft_consensus.cc:2241] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:51:59.989429 2526 raft_consensus.cc:2270] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:51:59.989861 2526 ts_tablet_manager.cc:1905] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:51:59.997231 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1dc6d6d2b1458cb57875d2e25354bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:51:59.997634 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c1dc6d6d2b1458cb57875d2e25354bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.000811 1154 tablet_bootstrap.cc:492] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.000928 1154 ts_tablet_manager.cc:1397] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20250629 01:52:00.001682 1154 raft_consensus.cc:357] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.001823 1154 raft_consensus.cc:383] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.001876 1154 raft_consensus.cc:738] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.002072 1154 consensus_queue.cc:260] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.002560 1154 ts_tablet_manager.cc:1428] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:00.002779 1154 tablet_bootstrap.cc:492] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.003337 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 40592f11f430469f9c8e82eba25b38ed (table default.Table_8e4f4f0655b44c68953073e4f0298fb8 [id=c0e46a0b48ee4ae58b819cd01a157a47]) successfully deleted
I20250629 01:52:00.009287 1155 tablet_bootstrap.cc:492] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.009394 1155 ts_tablet_manager.cc:1397] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.000s
I20250629 01:52:00.010010 1155 raft_consensus.cc:357] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.010166 1155 raft_consensus.cc:383] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.010218 1155 raft_consensus.cc:738] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.011745 1155 consensus_queue.cc:260] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.012053 1154 tablet_bootstrap.cc:654] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:00.015550 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43624b0c909647f49981fff8584c956f consensus state has no leader
I20250629 01:52:00.015755 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943] alter complete (version 5)
W20250629 01:52:00.016947 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 631ca7239f574441b37c11e104c3d56b consensus state has no leader
I20250629 01:52:00.017007 1155 ts_tablet_manager.cc:1428] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250629 01:52:00.017290 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d73c2eb0d69e467b94a250e1790d8142 consensus state has no leader
I20250629 01:52:00.017500 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_a51f9e473f8f426db01a3dbfd4303993 [id=55cb83afbc4f461f8946c50d83054bdb] alter complete (version 1)
I20250629 01:52:00.018424 2454 raft_consensus.cc:491] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.018535 2454 raft_consensus.cc:513] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.019013 2454 leader_election.cc:290] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:00.020211 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2845d8fbf3f34c769027b9b6bb3e1f08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:00.020319 850 leader_election.cc:343] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 1c995ce14c9a4966b842fcba520cf87d
I20250629 01:52:00.022176 2617 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a0b76160e22c46929936cb9682322fae)
I20250629 01:52:00.023255 1159 tablet_bootstrap.cc:492] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.023412 1159 ts_tablet_manager.cc:1397] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20250629 01:52:00.023587 2454 raft_consensus.cc:491] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.023695 2454 raft_consensus.cc:513] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.024134 2454 leader_election.cc:290] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:00.024060 1159 raft_consensus.cc:357] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.024257 1159 raft_consensus.cc:383] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.024302 1159 raft_consensus.cc:738] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.024461 1159 consensus_queue.cc:260] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.024845 1159 ts_tablet_manager.cc:1428] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:00.025146 1159 tablet_bootstrap.cc:492] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.022507 2617 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } }
I20250629 01:52:00.026517 2454 raft_consensus.cc:491] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.026602 2454 raft_consensus.cc:513] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.027020 2454 leader_election.cc:290] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:00.027096 850 leader_election.cc:343] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 2845d8fbf3f34c769027b9b6bb3e1f08
W20250629 01:52:00.027923 850 leader_election.cc:343] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: d6bf3c7d0bb548e0b77ba8b569a72d0d
I20250629 01:52:00.029462 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c5371085a4334fbfbd6f4e385058da15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.029897 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5371085a4334fbfbd6f4e385058da15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.030032 1155 tablet_bootstrap.cc:492] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.030810 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1100 } } ignore_live_leader: false dest_uuid: "a0b76160e22c46929936cb9682322fae" is_pre_election: true
I20250629 01:52:00.031420 1497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1100 } } ignore_live_leader: false dest_uuid: "f304077528b34c61857a659e5873631d" is_pre_election: true
I20250629 01:52:00.031754 1155 tablet_bootstrap.cc:654] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.032147 1969 leader_election.cc:304] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f4258c004e04bcf8216bb7c8fd56c54; no voters: a0b76160e22c46929936cb9682322fae, f304077528b34c61857a659e5873631d
I20250629 01:52:00.032538 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_346dcc9d7d4c43fb8bc4573429d3aaa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.032871 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_346dcc9d7d4c43fb8bc4573429d3aaa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.034049 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2249d5cba1544653b088312e7d74802e with delete_type TABLET_DATA_DELETED (replaced by tablet d387af53209147a6bac7afeda9a411ab) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.035629 451 master_service.cc:948] ReplaceTablet: received request to replace tablet e4ff12d702bf48039d63c437079979fd from {username='slave'} at 127.0.0.1:56238
W20250629 01:52:00.039743 850 leader_election.cc:343] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d6bf3c7d0bb548e0b77ba8b569a72d0d
I20250629 01:52:00.039832 850 leader_election.cc:304] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.041332 2558 tablet.cc:1722] T 9c799a04c0f74492b9fb674f9b0da240 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.045675 1159 tablet_bootstrap.cc:654] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.048605 2578 tablet.cc:1722] T 9c799a04c0f74492b9fb674f9b0da240 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.051062 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.051527 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 8975c1deaaf44761be791f460b53ec74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.052405 2454 raft_consensus.cc:2747] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:00.052726 850 leader_election.cc:343] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 1c995ce14c9a4966b842fcba520cf87d
I20250629 01:52:00.052806 850 leader_election.cc:304] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.052960 2454 raft_consensus.cc:2747] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:00.053082 850 leader_election.cc:343] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2845d8fbf3f34c769027b9b6bb3e1f08
I20250629 01:52:00.053193 850 leader_election.cc:304] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.053339 2454 raft_consensus.cc:2747] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.053638 2617 leader_election.cc:290] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a0b76160e22c46929936cb9682322fae (127.0.104.62:34747), f304077528b34c61857a659e5873631d (127.0.104.60:32789)
I20250629 01:52:00.053840 2617 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:52:00.056730 1154 tablet_bootstrap.cc:492] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.056878 1154 ts_tablet_manager.cc:1397] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.001s
I20250629 01:52:00.057518 1154 raft_consensus.cc:357] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.057700 1154 raft_consensus.cc:383] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.057780 1154 raft_consensus.cc:738] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.057982 1154 consensus_queue.cc:260] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.058495 1154 ts_tablet_manager.cc:1428] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:00.060230 1154 tablet_bootstrap.cc:492] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.061875 1154 tablet_bootstrap.cc:654] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.067337 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1217) took 1577 ms. Trace:
I20250629 01:52:00.067557 1158 rpcz_store.cc:276] 0629 01:51:58.490130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.490161 (+ 31us) service_pool.cc:225] Handling call
0629 01:51:59.892307 (+1402146us) ts_tablet_manager.cc:1145] Deleting tablet 40592f11f430469f9c8e82eba25b38ed
0629 01:51:59.892314 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:51:59.915961 (+ 23647us) tablet_metadata.cc:641] Metadata flushed
0629 01:51:59.983140 (+ 67179us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.067317 (+ 84177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402021}
I20250629 01:52:00.068172 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1218) took 1577 ms. Trace:
I20250629 01:52:00.068292 1158 rpcz_store.cc:276] 0629 01:51:58.490425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:58.526251 (+ 35826us) service_pool.cc:225] Handling call
0629 01:52:00.068039 (+1541788us) ts_tablet_manager.cc:1145] Deleting tablet 2a876b3db3904a4297b5b13a9325b1c3
0629 01:52:00.068047 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.068157 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541645}
I20250629 01:52:00.069017 1158 tablet_replica.cc:331] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.069197 2526 ts_tablet_manager.cc:1918] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.069278 1158 raft_consensus.cc:2241] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:00.069561 1158 pending_rounds.cc:70] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:00.069653 1158 pending_rounds.cc:77] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759220579016704 op_type: NO_OP noop_request { }
I20250629 01:52:00.070055 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 40592f11f430469f9c8e82eba25b38ed (table default.Table_8e4f4f0655b44c68953073e4f0298fb8 [id=c0e46a0b48ee4ae58b819cd01a157a47]) successfully deleted
W20250629 01:52:00.070343 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2a876b3db3904a4297b5b13a9325b1c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a876b3db3904a4297b5b13a9325b1c3 already in progress: creating tablet
I20250629 01:52:00.084347 1158 raft_consensus.cc:2887] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.084506 1158 raft_consensus.cc:2270] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.085233 1158 ts_tablet_manager.cc:1905] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.085585 1156 ts_tablet_manager.cc:1918] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.085733 1156 log.cc:1199] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/9fe9674ffeb44ba6ab60de59d4c08670
I20250629 01:52:00.086051 1156 ts_tablet_manager.cc:1939] T 9fe9674ffeb44ba6ab60de59d4c08670 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.086294 1154 tablet_bootstrap.cc:492] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.086465 1154 ts_tablet_manager.cc:1397] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s
I20250629 01:52:00.086927 1156 tablet_replica.cc:331] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:00.087141 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e4ff12d702bf48039d63c437079979fd
I20250629 01:52:00.087131 1156 raft_consensus.cc:2241] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.087101 1154 raft_consensus.cc:357] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.087230 1154 raft_consensus.cc:383] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.087330 1154 raft_consensus.cc:738] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.087347 1156 raft_consensus.cc:2270] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.087548 451 catalog_manager.cc:6652] ReplaceTablet: tablet e4ff12d702bf48039d63c437079979fd deleted and replaced by tablet dd7a6028e372422e88079618e748a2bf
I20250629 01:52:00.087585 1154 consensus_queue.cc:260] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.087770 1156 ts_tablet_manager.cc:1905] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.088847 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet e4ff12d702bf48039d63c437079979fd with delete_type TABLET_DATA_DELETED (replaced by tablet dd7a6028e372422e88079618e748a2bf) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.088997 742 tablet_service.cc:1515] Processing DeleteTablet for tablet e4ff12d702bf48039d63c437079979fd with delete_type TABLET_DATA_DELETED (replaced by tablet dd7a6028e372422e88079618e748a2bf) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.089664 2527 raft_consensus.cc:491] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.089646 1155 tablet_bootstrap.cc:492] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.089774 2527 raft_consensus.cc:513] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.089977 1155 ts_tablet_manager.cc:1397] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20250629 01:52:00.090395 2527 leader_election.cc:290] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.090863 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0782c7e652419096710563930f2072" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.091001 777 raft_consensus.cc:2466] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:00.091388 986 leader_election.cc:304] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.091672 2527 raft_consensus.cc:2802] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.091735 2527 raft_consensus.cc:491] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.091781 2527 raft_consensus.cc:3058] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.092047 1154 ts_tablet_manager.cc:1428] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:52:00.082789 912 raft_consensus.cc:3058] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.093230 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 9fe9674ffeb44ba6ab60de59d4c08670 (table default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]) successfully deleted
I20250629 01:52:00.093379 1159 tablet_bootstrap.cc:492] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.093509 1159 ts_tablet_manager.cc:1397] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.000s
I20250629 01:52:00.094079 1159 raft_consensus.cc:357] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.094215 1159 raft_consensus.cc:383] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.094276 1159 raft_consensus.cc:738] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.094579 1155 raft_consensus.cc:357] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.094697 1155 raft_consensus.cc:383] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.094734 1155 raft_consensus.cc:738] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.088258 892 tablet_service.cc:1515] Processing DeleteTablet for tablet e4ff12d702bf48039d63c437079979fd with delete_type TABLET_DATA_DELETED (replaced by tablet dd7a6028e372422e88079618e748a2bf) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.094925 1159 consensus_queue.cc:260] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.095237 2454 raft_consensus.cc:491] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.095244 1155 consensus_queue.cc:260] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.095340 2454 raft_consensus.cc:513] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.095817 2454 leader_election.cc:290] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:00.096532 850 leader_election.cc:343] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 2249d5cba1544653b088312e7d74802e
I20250629 01:52:00.090764 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0782c7e652419096710563930f2072" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:00.097178 988 leader_election.cc:343] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.098253 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_49b84436278348afa56cf408cab52a43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.098683 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49b84436278348afa56cf408cab52a43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.099928 1155 ts_tablet_manager.cc:1428] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250629 01:52:00.100615 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b9cb2b777214494e9a98a81d9b7785ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.101008 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9cb2b777214494e9a98a81d9b7785ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.101040 1155 tablet_bootstrap.cc:492] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.105935 1155 tablet_bootstrap.cc:654] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:00.106611 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 30265f664bff4b058e587edd9e0c5978 (table default.Table_75ca31dbf2c745918cc69f2166dd9b89 [id=43c1739e8c674274a89bbfc16f16e3c6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 30265f664bff4b058e587edd9e0c5978 consensus state has no leader
I20250629 01:52:00.115842 2526 log.cc:1199] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a6a012ba891949538800581436ef5380
I20250629 01:52:00.118081 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dee672b0788449d08349b1fecc061b43" } modify_external_catalogs: true
I20250629 01:52:00.118201 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dee672b0788449d08349b1fecc061b43" } modify_external_catalogs: true
I20250629 01:52:00.120723 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dc3b011b71e3400bafeb102d3c3cd403" } modify_external_catalogs: true
I20250629 01:52:00.120868 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dc3b011b71e3400bafeb102d3c3cd403" } modify_external_catalogs: true
I20250629 01:52:00.121309 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c2146a78ba3746aea6dc9d3614b1e5c7" } new_table_name: "default.Table_0246da0c3faf4867b6bb1bbf22977ed0" modify_external_catalogs: true
I20250629 01:52:00.123174 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a08dd1de5190465b9b899ba5eaf80fff" } modify_external_catalogs: true
I20250629 01:52:00.123283 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a08dd1de5190465b9b899ba5eaf80fff" } modify_external_catalogs: true
I20250629 01:52:00.123648 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e5ccc8748dbe46e697a27a8368252d15" } modify_external_catalogs: true
I20250629 01:52:00.123741 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e5ccc8748dbe46e697a27a8368252d15" } modify_external_catalogs: true
I20250629 01:52:00.124583 1155 tablet_bootstrap.cc:492] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.124687 1155 ts_tablet_manager.cc:1397] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250629 01:52:00.125252 1155 raft_consensus.cc:357] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:00.127282 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 421ffa84a7664826a6cbc303653fd00b consensus state has no leader
I20250629 01:52:00.130472 2534 raft_consensus.cc:491] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.130576 2534 raft_consensus.cc:513] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.130861 2526 ts_tablet_manager.cc:1939] T a6a012ba891949538800581436ef5380 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.131031 2534 leader_election.cc:290] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.131245 2526 tablet_replica.cc:331] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:00.131508 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a6a012ba891949538800581436ef5380 (table default.Table_e69e519c24094e298cdcd58c4567f525 [id=ad7989043c924e598bdc640d4deecec3]) successfully deleted
I20250629 01:52:00.134156 1154 tablet_bootstrap.cc:492] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.136950 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d87639255ec46d98fcfb0f9638ee39a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.137104 1051 raft_consensus.cc:2466] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:00.137495 710 leader_election.cc:304] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.137770 2548 raft_consensus.cc:2802] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.137825 2548 raft_consensus.cc:491] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.137871 2548 raft_consensus.cc:3058] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.140235 2548 raft_consensus.cc:513] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.140511 1154 tablet_bootstrap.cc:654] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.140662 2548 leader_election.cc:290] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.142328 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d87639255ec46d98fcfb0f9638ee39a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:00.142442 1051 raft_consensus.cc:3058] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.125385 1155 raft_consensus.cc:383] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.146910 1155 raft_consensus.cc:738] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.147210 1155 consensus_queue.cc:260] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.147377 2527 raft_consensus.cc:513] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.148707 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0782c7e652419096710563930f2072" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:00.148838 777 raft_consensus.cc:3058] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.148957 1155 ts_tablet_manager.cc:1428] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250629 01:52:00.148597 1156 ts_tablet_manager.cc:1918] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.149057 1156 log.cc:1199] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/45ca747f03524076b4112e4db78614fe
I20250629 01:52:00.149362 1156 ts_tablet_manager.cc:1939] T 45ca747f03524076b4112e4db78614fe P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.151079 777 raft_consensus.cc:2466] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:00.151609 986 leader_election.cc:304] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.152153 2580 raft_consensus.cc:2802] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.152238 2580 raft_consensus.cc:695] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:00.152400 2580 consensus_queue.cc:237] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.132180 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d87639255ec46d98fcfb0f9638ee39a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.153504 2526 raft_consensus.cc:2241] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:52:00.153774 712 leader_election.cc:343] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.133015 850 leader_election.cc:343] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 2249d5cba1544653b088312e7d74802e
I20250629 01:52:00.142045 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d87639255ec46d98fcfb0f9638ee39a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:00.146768 1159 ts_tablet_manager.cc:1428] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.053s user 0.002s sys 0.002s
I20250629 01:52:00.154035 850 leader_election.cc:304] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.154235 1159 tablet_bootstrap.cc:492] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:00.154645 712 leader_election.cc:343] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.155126 2626 raft_consensus.cc:2747] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.155581 1159 tablet_bootstrap.cc:654] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.156376 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 45ca747f03524076b4112e4db78614fe (table default.Table_e39f46841d454041bb064e74d3c27bc0 [id=f7eba54966074342a3632852c544ff35]) successfully deleted
I20250629 01:52:00.156723 1156 tablet_replica.cc:331] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:00.156850 2527 leader_election.cc:290] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.157024 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b0782c7e652419096710563930f2072" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:00.157395 988 leader_election.cc:343] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.157155 1156 raft_consensus.cc:2241] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:00.157763 1051 raft_consensus.cc:2466] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:00.157810 1156 pending_rounds.cc:70] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:00.158106 710 leader_election.cc:304] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:00.158885 2548 raft_consensus.cc:2802] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.159165 2548 raft_consensus.cc:695] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:00.159463 2548 consensus_queue.cc:237] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.164005 1156 pending_rounds.cc:77] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759222480072704 op_type: NO_OP noop_request { }
I20250629 01:52:00.164206 1156 raft_consensus.cc:2887] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.164353 1156 raft_consensus.cc:2270] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.165176 1156 ts_tablet_manager.cc:1905] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.167284 1158 ts_tablet_manager.cc:1918] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:00.153970 2526 raft_consensus.cc:2270] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.168702 2526 ts_tablet_manager.cc:1905] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.170800 1156 ts_tablet_manager.cc:1918] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:00.172549 1156 log.cc:1199] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f3bdc8bda8a940be865679a05b90bd08
I20250629 01:52:00.172873 1155 tablet_bootstrap.cc:492] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.173008 1156 ts_tablet_manager.cc:1939] T f3bdc8bda8a940be865679a05b90bd08 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.174078 1156 tablet_replica.cc:331] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:00.174199 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f3bdc8bda8a940be865679a05b90bd08 (table default.Table_44f5d1913b0e4eedaac117acfdf02c22 [id=b6e2a7a8ccc245899af18bfd45e0f740]) successfully deleted
I20250629 01:52:00.174989 1156 raft_consensus.cc:2241] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.175122 1156 raft_consensus.cc:2270] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.175588 1156 ts_tablet_manager.cc:1905] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.177244 1155 tablet_bootstrap.cc:654] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.180579 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_001859f907034195a0b9358f6fff60c7" } modify_external_catalogs: true
I20250629 01:52:00.180742 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_001859f907034195a0b9358f6fff60c7" } modify_external_catalogs: true
I20250629 01:52:00.181630 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7788acac435247629f3a62bc49aa2ca0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.182157 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7788acac435247629f3a62bc49aa2ca0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.187646 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ad8669ae4f784da9b23f7890936e487d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.188115 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad8669ae4f784da9b23f7890936e487d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.188117 2487 tablet.cc:1722] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.194972 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]). Expected version 2 got 0
I20250629 01:52:00.195840 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:52:00.203045 1158 log.cc:1199] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/3a81e6f9711c4d398ae09f33a75566ae
I20250629 01:52:00.203459 1158 ts_tablet_manager.cc:1939] T 3a81e6f9711c4d398ae09f33a75566ae P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.204005 2553 raft_consensus.cc:491] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.204120 2553 raft_consensus.cc:513] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.204574 2553 leader_election.cc:290] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.205051 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe0cd4b8f7d47268a80f645bc8b2b9d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.205197 1051 raft_consensus.cc:2466] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:00.205391 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:52:00.205487 710 leader_election.cc:304] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.205682 2553 raft_consensus.cc:2802] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.205730 2553 raft_consensus.cc:491] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.205777 2553 raft_consensus.cc:3058] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.208003 2553 raft_consensus.cc:513] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.208528 2553 leader_election.cc:290] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.208948 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe0cd4b8f7d47268a80f645bc8b2b9d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:00.209081 1051 raft_consensus.cc:3058] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.209285 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe0cd4b8f7d47268a80f645bc8b2b9d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:00.209645 712 leader_election.cc:343] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.211338 1051 raft_consensus.cc:2466] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:00.211640 710 leader_election.cc:304] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:00.212426 1154 tablet_bootstrap.cc:492] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.212534 1154 ts_tablet_manager.cc:1397] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.079s user 0.006s sys 0.005s
I20250629 01:52:00.213143 1154 raft_consensus.cc:357] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.213276 1154 raft_consensus.cc:383] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.213322 1154 raft_consensus.cc:738] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.213500 1154 consensus_queue.cc:260] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.213881 1154 ts_tablet_manager.cc:1428] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:00.214254 2548 raft_consensus.cc:2802] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.214797 1155 tablet_bootstrap.cc:492] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.214794 2548 raft_consensus.cc:695] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:00.215119 2548 consensus_queue.cc:237] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.216027 1154 tablet_bootstrap.cc:492] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.214916 1155 ts_tablet_manager.cc:1397] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250629 01:52:00.217491 1154 tablet_bootstrap.cc:654] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.218484 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1fe0cd4b8f7d47268a80f645bc8b2b9d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:00.218874 712 leader_election.cc:343] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.219885 1156 ts_tablet_manager.cc:1918] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.221621 1156 log.cc:1199] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:52:00.222012 1156 ts_tablet_manager.cc:1939] T daf312e60c014fe698d3ec9f1a1b769c P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.222807 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_94f201340df34d00a02cde31f001d6d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:00.223021 1156 tablet_replica.cc:331] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:00.223101 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet daf312e60c014fe698d3ec9f1a1b769c (table default.Table_3f1b07c24c6441f4a4965f8e055648fe [id=df4a1725208848efa85ea19a1d3da531]) successfully deleted
I20250629 01:52:00.223176 1156 raft_consensus.cc:2241] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:00.223341 1156 pending_rounds.cc:70] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:00.223414 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a3f4932fdc864c30a60546170db190a9" } new_table_name: "default.Table_d3ef62e8776f4695b4b62cbeb32c19fd" modify_external_catalogs: true
I20250629 01:52:00.224787 1155 raft_consensus.cc:357] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.225104 1155 raft_consensus.cc:383] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.225373 1155 raft_consensus.cc:738] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.225926 1155 consensus_queue.cc:260] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.227368 1155 ts_tablet_manager.cc:1428] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250629 01:52:00.228577 1155 tablet_bootstrap.cc:492] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.229015 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3181bf94172b4d6f95360a4c1be43602"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:51:59.104570 2017 rpcz_store.cc:276] 0629 01:51:52.582653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:52.582714 (+ 61us) service_pool.cc:225] Handling call
0629 01:51:58.236146 (+5653432us) raft_consensus.cc:1469] Updating replica for 339 ops
0629 01:51:58.264287 (+ 28141us) raft_consensus.cc:1532] Early marking committed up to index 0
0629 01:51:58.264295 (+ 8us) raft_consensus.cc:1537] Triggering prepare for 339 ops
0629 01:51:58.265134 (+ 839us) write_op.cc:270] Start()
0629 01:51:58.265191 (+ 57us) write_op.cc:276] Timestamp: P: 1751161907698101 usec, L: 0
0629 01:51:58.311915 (+ 46724us) write_op.cc:270] Start()
0629 01:51:58.311945 (+ 30us) write_op.cc:276] Timestamp: P: 1751161907698569 usec, L: 0
0629 01:51:58.312416 (+ 471us) write_op.cc:270] Start()
0629 01:51:58.312449 (+ 33us) write_op.cc:276] Timestamp: P: 1751161907711424 usec, L: 0
0629 01:51:58.312608 (+ 159us) write_op.cc:270] Start()
0629 01:51:58.312625 (+ 17us) write_op.cc:276] Timestamp: P: 1751161907727432 usec, L: 0
0629 01:51:58.312763 (+ 138us) write_op.cc:270] Start()
0629 01:51:58.312779 (+ 16us) write_op.cc:276] Timestamp: P: 1751161907737149 usec, L: 0
0629 01:51:58.312905 (+ 126us) write_op.cc:270] Start()
0629 01:51:58.312922 (+ 17us) write_op.cc:276] Timestamp: P: 1751161907737829 usec, L: 0
0629 01:51:58.313047 (+ 125us) write_op.cc:270] Start()
0629 01:51:58.313089 (+ 42us) write_op.cc:276] Timestamp: P: 1751161907738501 usec, L: 0
0629 01:51:58.313239 (+ 150us) write_op.cc:270] Start()
0629 01:51:58.313256 (+ 17us) write_op.cc:276] Timestamp: P: 1751161907768742 usec, L: 0
0629 01:51:58.313377 (+ 121us) write_op.cc:270] Start()
0629 01:51:58.313395 (+ 18us) write_op.cc:276] Timestamp: P: 1751161907769503 usec, L: 0
0629 01:51:58.313505 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.313520 (+ 15us) write_op.cc:276] Timestamp: P: 1751161907771097 usec, L: 0
0629 01:51:58.313685 (+ 165us) write_op.cc:270] Start()
0629 01:51:58.313704 (+ 19us) write_op.cc:276] Timestamp: P: 1751161907772722 usec, L: 0
0629 01:51:58.313820 (+ 116us) write_op.cc:270] Start()
0629 01:51:58.313838 (+ 18us) write_op.cc:276] Timestamp: P: 1751161907778118 usec, L: 0
0629 01:51:58.313940 (+ 102us) write_op.cc:270] Start()
0629 01:51:58.313956 (+ 16us) write_op.cc:276] Timestamp: P: 1751161907962613 usec, L: 0
0629 01:51:58.314107 (+ 151us) write_op.cc:270] Start()
0629 01:51:58.314124 (+ 17us) write_op.cc:276] Timestamp: P: 1751161907963103 usec, L: 0
0629 01:51:58.314269 (+ 145us) write_op.cc:270] Start()
0629 01:51:58.314287 (+ 18us) write_op.cc:276] Timestamp: P: 1751161907965731 usec, L: 0
0629 01:51:58.314392 (+ 105us) write_op.cc:270] Start()
0629 01:51:58.314407 (+ 15us) write_op.cc:276] Timestamp: P: 1751161907966264 usec, L: 0
0629 01:51:58.314514 (+ 107us) write_op.cc:270] Start()
0629 01:51:58.314540 (+ 26us) write_op.cc:276] Timestamp: P: 1751161907968963 usec, L: 0
0629 01:51:58.314651 (+ 111us) write_op.cc:270] Start()
0629 01:51:58.314678 (+ 27us) write_op.cc:276] Timestamp: P: 1751161907998040 usec, L: 0
0629 01:51:58.314779 (+ 101us) write_op.cc:270] Start()
0629 01:51:58.314793 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908031386 usec, L: 0
0629 01:51:58.314897 (+ 104us) write_op.cc:270] Start()
0629 01:51:58.314912 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908037165 usec, L: 0
0629 01:51:58.315033 (+ 121us) write_op.cc:270] Start()
0629 01:51:58.315048 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908037710 usec, L: 0
0629 01:51:58.315155 (+ 107us) write_op.cc:270] Start()
0629 01:51:58.315170 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908038217 usec, L: 0
0629 01:51:58.315260 (+ 90us) write_op.cc:270] Start()
0629 01:51:58.315273 (+ 13us) write_op.cc:276] Timestamp: P: 1751161908038723 usec, L: 0
0629 01:51:58.315371 (+ 98us) write_op.cc:270] Start()
0629 01:51:58.315386 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908068459 usec, L: 0
0629 01:51:58.315542 (+ 156us) write_op.cc:270] Start()
0629 01:51:58.315560 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908085668 usec, L: 0
0629 01:51:58.325910 (+ 10350us) write_op.cc:270] Start()
0629 01:51:58.325936 (+ 26us) write_op.cc:276] Timestamp: P: 1751161908086701 usec, L: 0
0629 01:51:58.326109 (+ 173us) write_op.cc:270] Start()
0629 01:51:58.326129 (+ 20us) write_op.cc:276] Timestamp: P: 1751161908087469 usec, L: 0
0629 01:51:58.341915 (+ 15786us) write_op.cc:270] Start()
0629 01:51:58.341948 (+ 33us) write_op.cc:276] Timestamp: P: 1751161908088233 usec, L: 0
0629 01:51:58.342124 (+ 176us) write_op.cc:270] Start()
0629 01:51:58.342147 (+ 23us) write_op.cc:276] Timestamp: P: 1751161908088953 usec, L: 0
0629 01:51:58.342290 (+ 143us) write_op.cc:270] Start()
0629 01:51:58.342333 (+ 43us) write_op.cc:276] Timestamp: P: 1751161908089613 usec, L: 0
0629 01:51:58.342542 (+ 209us) write_op.cc:270] Start()
0629 01:51:58.342561 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908090301 usec, L: 0
0629 01:51:58.342675 (+ 114us) write_op.cc:270] Start()
0629 01:51:58.342691 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908096977 usec, L: 0
0629 01:51:58.342806 (+ 115us) write_op.cc:270] Start()
0629 01:51:58.342821 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908189779 usec, L: 0
0629 01:51:58.342926 (+ 105us) write_op.cc:270] Start()
0629 01:51:58.342943 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908190311 usec, L: 0
0629 01:51:58.343058 (+ 115us) write_op.cc:270] Start()
0629 01:51:58.343076 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908210091 usec, L: 0
0629 01:51:58.343195 (+ 119us) write_op.cc:270] Start()
0629 01:51:58.343211 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908210877 usec, L: 0
0629 01:51:58.343324 (+ 113us) write_op.cc:270] Start()
0629 01:51:58.343349 (+ 25us) write_op.cc:276] Timestamp: P: 1751161908213305 usec, L: 0
0629 01:51:58.343468 (+ 119us) write_op.cc:270] Start()
0629 01:51:58.343484 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908251461 usec, L: 0
0629 01:51:58.343596 (+ 112us) write_op.cc:270] Start()
0629 01:51:58.343614 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908266253 usec, L: 0
0629 01:51:58.343716 (+ 102us) write_op.cc:270] Start()
0629 01:51:58.343733 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908267431 usec, L: 0
0629 01:51:58.343847 (+ 114us) write_op.cc:270] Start()
0629 01:51:58.343863 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908283064 usec, L: 0
0629 01:51:58.343992 (+ 129us) write_op.cc:270] Start()
0629 01:51:58.344015 (+ 23us) write_op.cc:276] Timestamp: P: 1751161908284453 usec, L: 0
0629 01:51:58.344147 (+ 132us) write_op.cc:270] Start()
0629 01:51:58.344165 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908292366 usec, L: 0
0629 01:51:58.344295 (+ 130us) write_op.cc:270] Start()
0629 01:51:58.344312 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908292853 usec, L: 0
0629 01:51:58.344473 (+ 161us) write_op.cc:270] Start()
0629 01:51:58.344492 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908295281 usec, L: 0
0629 01:51:58.344610 (+ 118us) write_op.cc:270] Start()
0629 01:51:58.344626 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908295999 usec, L: 0
0629 01:51:58.344745 (+ 119us) write_op.cc:270] Start()
0629 01:51:58.344762 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908352591 usec, L: 0
0629 01:51:58.344871 (+ 109us) write_op.cc:270] Start()
0629 01:51:58.344888 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908376629 usec, L: 0
0629 01:51:58.345016 (+ 128us) write_op.cc:270] Start()
0629 01:51:58.345034 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908387848 usec, L: 0
0629 01:51:58.350391 (+ 5357us) write_op.cc:270] Start()
0629 01:51:58.350415 (+ 24us) write_op.cc:276] Timestamp: P: 1751161908388428 usec, L: 0
0629 01:51:58.350626 (+ 211us) write_op.cc:270] Start()
0629 01:51:58.350645 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908399676 usec, L: 0
0629 01:51:58.350776 (+ 131us) write_op.cc:270] Start()
0629 01:51:58.350793 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908405060 usec, L: 0
0629 01:51:58.350891 (+ 98us) write_op.cc:270] Start()
0629 01:51:58.350906 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908405489 usec, L: 0
0629 01:51:58.351032 (+ 126us) write_op.cc:270] Start()
0629 01:51:58.351054 (+ 22us) write_op.cc:276] Timestamp: P: 1751161908405950 usec, L: 0
0629 01:51:58.351158 (+ 104us) write_op.cc:270] Start()
0629 01:51:58.351173 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908416867 usec, L: 0
0629 01:51:58.351286 (+ 113us) write_op.cc:270] Start()
0629 01:51:58.351302 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908440597 usec, L: 0
0629 01:51:58.351405 (+ 103us) write_op.cc:270] Start()
0629 01:51:58.351420 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908448839 usec, L: 0
0629 01:51:58.351532 (+ 112us) write_op.cc:270] Start()
0629 01:51:58.351548 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908449555 usec, L: 0
0629 01:51:58.351647 (+ 99us) write_op.cc:270] Start()
0629 01:51:58.351662 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908464928 usec, L: 0
0629 01:51:58.351759 (+ 97us) write_op.cc:270] Start()
0629 01:51:58.351790 (+ 31us) write_op.cc:276] Timestamp: P: 1751161908480935 usec, L: 0
0629 01:51:58.351894 (+ 104us) write_op.cc:270] Start()
0629 01:51:58.351909 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908486856 usec, L: 0
0629 01:51:58.352023 (+ 114us) write_op.cc:270] Start()
0629 01:51:58.352038 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908499672 usec, L: 0
0629 01:51:58.352163 (+ 125us) write_op.cc:270] Start()
0629 01:51:58.352179 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908500104 usec, L: 0
0629 01:51:58.352277 (+ 98us) write_op.cc:270] Start()
0629 01:51:58.352291 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908500756 usec, L: 0
0629 01:51:58.352422 (+ 131us) write_op.cc:270] Start()
0629 01:51:58.352440 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908514782 usec, L: 0
0629 01:51:58.352553 (+ 113us) write_op.cc:270] Start()
0629 01:51:58.352572 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908521504 usec, L: 0
0629 01:51:58.352695 (+ 123us) write_op.cc:270] Start()
0629 01:51:58.352715 (+ 20us) write_op.cc:276] Timestamp: P: 1751161908528516 usec, L: 0
0629 01:51:58.355836 (+ 3121us) write_op.cc:270] Start()
0629 01:51:58.355856 (+ 20us) write_op.cc:276] Timestamp: P: 1751161908538967 usec, L: 0
0629 01:51:58.355998 (+ 142us) write_op.cc:270] Start()
0629 01:51:58.356015 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908574176 usec, L: 0
0629 01:51:58.356141 (+ 126us) write_op.cc:270] Start()
0629 01:51:58.356156 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908598220 usec, L: 0
0629 01:51:58.356263 (+ 107us) write_op.cc:270] Start()
0629 01:51:58.356278 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908601675 usec, L: 0
0629 01:51:58.356375 (+ 97us) write_op.cc:270] Start()
0629 01:51:58.356389 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908626419 usec, L: 0
0629 01:51:58.356514 (+ 125us) write_op.cc:270] Start()
0629 01:51:58.356533 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908632698 usec, L: 0
0629 01:51:58.356662 (+ 129us) write_op.cc:270] Start()
0629 01:51:58.356679 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908633139 usec, L: 0
0629 01:51:58.356775 (+ 96us) write_op.cc:270] Start()
0629 01:51:58.356789 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908633602 usec, L: 0
0629 01:51:58.356899 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.356913 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908667265 usec, L: 0
0629 01:51:58.357027 (+ 114us) write_op.cc:270] Start()
0629 01:51:58.357041 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908669194 usec, L: 0
0629 01:51:58.357153 (+ 112us) write_op.cc:270] Start()
0629 01:51:58.357167 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908670285 usec, L: 0
0629 01:51:58.357267 (+ 100us) write_op.cc:270] Start()
0629 01:51:58.357310 (+ 43us) write_op.cc:276] Timestamp: P: 1751161908741000 usec, L: 0
0629 01:51:58.357420 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.357435 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908741895 usec, L: 0
0629 01:51:58.357534 (+ 99us) write_op.cc:270] Start()
0629 01:51:58.357551 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908742425 usec, L: 0
0629 01:51:58.357651 (+ 100us) write_op.cc:270] Start()
0629 01:51:58.357665 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908743498 usec, L: 0
0629 01:51:58.357760 (+ 95us) write_op.cc:270] Start()
0629 01:51:58.357774 (+ 14us) write_op.cc:276] Timestamp: P: 1751161908747092 usec, L: 0
0629 01:51:58.357930 (+ 156us) write_op.cc:270] Start()
0629 01:51:58.357957 (+ 27us) write_op.cc:276] Timestamp: P: 1751161908747523 usec, L: 0
0629 01:51:58.358111 (+ 154us) write_op.cc:270] Start()
0629 01:51:58.358129 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908747972 usec, L: 0
0629 01:51:58.358268 (+ 139us) write_op.cc:270] Start()
0629 01:51:58.358285 (+ 17us) write_op.cc:276] Timestamp: P: 1751161908814748 usec, L: 0
0629 01:51:58.358390 (+ 105us) write_op.cc:270] Start()
0629 01:51:58.358409 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908815747 usec, L: 0
0629 01:51:58.358536 (+ 127us) write_op.cc:270] Start()
0629 01:51:58.358551 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908816528 usec, L: 0
0629 01:51:58.358674 (+ 123us) write_op.cc:270] Start()
0629 01:51:58.358695 (+ 21us) write_op.cc:276] Timestamp: P: 1751161908817234 usec, L: 0
0629 01:51:58.358810 (+ 115us) write_op.cc:270] Start()
0629 01:51:58.358831 (+ 21us) write_op.cc:276] Timestamp: P: 1751161908817916 usec, L: 0
0629 01:51:58.358941 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.358961 (+ 20us) write_op.cc:276] Timestamp: P: 1751161908847719 usec, L: 0
0629 01:51:58.359085 (+ 124us) write_op.cc:270] Start()
0629 01:51:58.359101 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908853277 usec, L: 0
0629 01:51:58.359206 (+ 105us) write_op.cc:270] Start()
0629 01:51:58.359221 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908866139 usec, L: 0
0629 01:51:58.359339 (+ 118us) write_op.cc:270] Start()
0629 01:51:58.359355 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908866673 usec, L: 0
0629 01:51:58.359472 (+ 117us) write_op.cc:270] Start()
0629 01:51:58.359492 (+ 20us) write_op.cc:276] Timestamp: P: 1751161908871902 usec, L: 0
0629 01:51:58.359638 (+ 146us) write_op.cc:270] Start()
0629 01:51:58.359654 (+ 16us) write_op.cc:276] Timestamp: P: 1751161908876519 usec, L: 0
0629 01:51:58.359764 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.359783 (+ 19us) write_op.cc:276] Timestamp: P: 1751161908882187 usec, L: 0
0629 01:51:58.359916 (+ 133us) write_op.cc:270] Start()
0629 01:51:58.359937 (+ 21us) write_op.cc:276] Timestamp: P: 1751161908885604 usec, L: 0
0629 01:51:58.360064 (+ 127us) write_op.cc:270] Start()
0629 01:51:58.360079 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908948460 usec, L: 0
0629 01:51:58.360215 (+ 136us) write_op.cc:270] Start()
0629 01:51:58.360233 (+ 18us) write_op.cc:276] Timestamp: P: 1751161908952781 usec, L: 0
0629 01:51:58.360369 (+ 136us) write_op.cc:270] Start()
0629 01:51:58.360384 (+ 15us) write_op.cc:276] Timestamp: P: 1751161908958153 usec, L: 0
0629 01:51:58.360480 (+ 96us) write_op.cc:270] Start()
0629 01:51:58.360494 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909010826 usec, L: 0
0629 01:51:58.360615 (+ 121us) write_op.cc:270] Start()
0629 01:51:58.360630 (+ 15us) write_op.cc:276] Timestamp: P: 1751161909024031 usec, L: 0
0629 01:51:58.360721 (+ 91us) write_op.cc:270] Start()
0629 01:51:58.360735 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909024620 usec, L: 0
0629 01:51:58.360830 (+ 95us) write_op.cc:270] Start()
0629 01:51:58.360843 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909038676 usec, L: 0
0629 01:51:58.360942 (+ 99us) write_op.cc:270] Start()
0629 01:51:58.360955 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909039158 usec, L: 0
0629 01:51:58.361058 (+ 103us) write_op.cc:270] Start()
0629 01:51:58.361072 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909039672 usec, L: 0
0629 01:51:58.361182 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.361196 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909060613 usec, L: 0
0629 01:51:58.361322 (+ 126us) write_op.cc:270] Start()
0629 01:51:58.361337 (+ 15us) write_op.cc:276] Timestamp: P: 1751161909061075 usec, L: 0
0629 01:51:58.361432 (+ 95us) write_op.cc:270] Start()
0629 01:51:58.361445 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909061557 usec, L: 0
0629 01:51:58.361536 (+ 91us) write_op.cc:270] Start()
0629 01:51:58.361550 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909072566 usec, L: 0
0629 01:51:58.361643 (+ 93us) write_op.cc:270] Start()
0629 01:51:58.361656 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909073327 usec, L: 0
0629 01:51:58.361741 (+ 85us) write_op.cc:270] Start()
0629 01:51:58.361754 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909073798 usec, L: 0
0629 01:51:58.361855 (+ 101us) write_op.cc:270] Start()
0629 01:51:58.361869 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909076062 usec, L: 0
0629 01:51:58.361961 (+ 92us) write_op.cc:270] Start()
0629 01:51:58.361984 (+ 23us) write_op.cc:276] Timestamp: P: 1751161909085801 usec, L: 0
0629 01:51:58.362072 (+ 88us) write_op.cc:270] Start()
0629 01:51:58.362084 (+ 12us) write_op.cc:276] Timestamp: P: 1751161909128245 usec, L: 0
0629 01:51:58.362178 (+ 94us) write_op.cc:270] Start()
0629 01:51:58.362192 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909133656 usec, L: 0
0629 01:51:58.362331 (+ 139us) write_op.cc:270] Start()
0629 01:51:58.362348 (+ 17us) write_op.cc:276] Timestamp: P: 1751161909152860 usec, L: 0
0629 01:51:58.362453 (+ 105us) write_op.cc:270] Start()
0629 01:51:58.362467 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909153397 usec, L: 0
0629 01:51:58.362577 (+ 110us) write_op.cc:270] Start()
0629 01:51:58.362592 (+ 15us) write_op.cc:276] Timestamp: P: 1751161909153831 usec, L: 0
0629 01:51:58.362678 (+ 86us) write_op.cc:270] Start()
0629 01:51:58.362696 (+ 18us) write_op.cc:276] Timestamp: P: 1751161909158736 usec, L: 0
0629 01:51:58.362812 (+ 116us) write_op.cc:270] Start()
0629 01:51:58.362825 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909159150 usec, L: 0
0629 01:51:58.362911 (+ 86us) write_op.cc:270] Start()
0629 01:51:58.370210 (+ 7299us) write_op.cc:276] Timestamp: P: 1751161909162095 usec, L: 0
0629 01:51:58.370365 (+ 155us) write_op.cc:270] Start()
0629 01:51:58.370384 (+ 19us) write_op.cc:276] Timestamp: P: 1751161909191549 usec, L: 0
0629 01:51:58.370493 (+ 109us) write_op.cc:270] Start()
0629 01:51:58.370508 (+ 15us) write_op.cc:276] Timestamp: P: 1751161909194389 usec, L: 0
0629 01:51:58.370604 (+ 96us) write_op.cc:270] Start()
0629 01:51:58.370621 (+ 17us) write_op.cc:276] Timestamp: P: 1751161909194838 usec, L: 0
0629 01:51:58.370736 (+ 115us) write_op.cc:270] Start()
0629 01:51:58.370750 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909270250 usec, L: 0
0629 01:51:58.370841 (+ 91us) write_op.cc:270] Start()
0629 01:51:58.370854 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909273378 usec, L: 0
0629 01:51:58.370962 (+ 108us) write_op.cc:270] Start()
0629 01:51:58.370987 (+ 25us) write_op.cc:276] Timestamp: P: 1751161909273879 usec, L: 0
0629 01:51:58.371093 (+ 106us) write_op.cc:270] Start()
0629 01:51:58.371114 (+ 21us) write_op.cc:276] Timestamp: P: 1751161909311120 usec, L: 0
0629 01:51:58.371242 (+ 128us) write_op.cc:270] Start()
0629 01:51:58.371274 (+ 32us) write_op.cc:276] Timestamp: P: 1751161909311833 usec, L: 0
0629 01:51:58.371360 (+ 86us) write_op.cc:270] Start()
0629 01:51:58.371373 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909384755 usec, L: 0
0629 01:51:58.371462 (+ 89us) write_op.cc:270] Start()
0629 01:51:58.371479 (+ 17us) write_op.cc:276] Timestamp: P: 1751161909396513 usec, L: 0
0629 01:51:58.371556 (+ 77us) write_op.cc:270] Start()
0629 01:51:58.371569 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909397286 usec, L: 0
0629 01:51:58.371659 (+ 90us) write_op.cc:270] Start()
0629 01:51:58.371672 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909402111 usec, L: 0
0629 01:51:58.371760 (+ 88us) write_op.cc:270] Start()
0629 01:51:58.371773 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909402630 usec, L: 0
0629 01:51:58.371858 (+ 85us) write_op.cc:270] Start()
0629 01:51:58.371870 (+ 12us) write_op.cc:276] Timestamp: P: 1751161909403189 usec, L: 0
0629 01:51:58.371979 (+ 109us) write_op.cc:270] Start()
0629 01:51:58.371993 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909403632 usec, L: 0
0629 01:51:58.372078 (+ 85us) write_op.cc:270] Start()
0629 01:51:58.372111 (+ 33us) write_op.cc:276] Timestamp: P: 1751161909415863 usec, L: 0
0629 01:51:58.372222 (+ 111us) write_op.cc:270] Start()
0629 01:51:58.372238 (+ 16us) write_op.cc:276] Timestamp: P: 1751161909416344 usec, L: 0
0629 01:51:58.372338 (+ 100us) write_op.cc:270] Start()
0629 01:51:58.372351 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909416816 usec, L: 0
0629 01:51:58.372459 (+ 108us) write_op.cc:270] Start()
0629 01:51:58.372472 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909419962 usec, L: 0
0629 01:51:58.372571 (+ 99us) write_op.cc:270] Start()
0629 01:51:58.372584 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909426340 usec, L: 0
0629 01:51:58.372686 (+ 102us) write_op.cc:270] Start()
0629 01:51:58.372701 (+ 15us) write_op.cc:276] Timestamp: P: 1751161909426783 usec, L: 0
0629 01:51:58.372783 (+ 82us) write_op.cc:270] Start()
0629 01:51:58.372804 (+ 21us) write_op.cc:276] Timestamp: P: 1751161909427285 usec, L: 0
0629 01:51:58.372885 (+ 81us) write_op.cc:270] Start()
0629 01:51:58.372897 (+ 12us) write_op.cc:276] Timestamp: P: 1751161909427704 usec, L: 0
0629 01:51:58.372989 (+ 92us) write_op.cc:270] Start()
0629 01:51:58.373001 (+ 12us) write_op.cc:276] Timestamp: P: 1751161909786020 usec, L: 0
0629 01:51:58.373133 (+ 132us) write_op.cc:270] Start()
0629 01:51:58.373149 (+ 16us) write_op.cc:276] Timestamp: P: 1751161909788331 usec, L: 0
0629 01:51:58.373255 (+ 106us) write_op.cc:270] Start()
0629 01:51:58.373268 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909789113 usec, L: 0
0629 01:51:58.373348 (+ 80us) write_op.cc:270] Start()
0629 01:51:58.373360 (+ 12us) write_op.cc:276] Timestamp: P: 1751161909789577 usec, L: 0
0629 01:51:58.373446 (+ 86us) write_op.cc:270] Start()
0629 01:51:58.373458 (+ 12us) write_op.cc:276] Timestamp: P: 1751161909790477 usec, L: 0
0629 01:51:58.373562 (+ 104us) write_op.cc:270] Start()
0629 01:51:58.373575 (+ 13us) write_op.cc:276] Timestamp: P: 1751161909793292 usec, L: 0
0629 01:51:58.408865 (+ 35290us) write_op.cc:270] Start()
0629 01:51:58.408909 (+ 44us) write_op.cc:276] Timestamp: P: 1751161909798167 usec, L: 0
0629 01:51:58.409178 (+ 269us) write_op.cc:270] Start()
0629 01:51:58.409205 (+ 27us) write_op.cc:276] Timestamp: P: 1751161909800195 usec, L: 0
0629 01:51:58.409356 (+ 151us) write_op.cc:270] Start()
0629 01:51:58.409377 (+ 21us) write_op.cc:276] Timestamp: P: 1751161909800757 usec, L: 0
0629 01:51:58.410625 (+ 1248us) write_op.cc:270] Start()
0629 01:51:58.410654 (+ 29us) write_op.cc:276] Timestamp: P: 1751161909801235 usec, L: 0
0629 01:51:58.410814 (+ 160us) write_op.cc:270] Start()
0629 01:51:58.410834 (+ 20us) write_op.cc:276] Timestamp: P: 1751161909801757 usec, L: 0
0629 01:51:58.410955 (+ 121us) write_op.cc:270] Start()
0629 01:51:58.410973 (+ 18us) write_op.cc:276] Timestamp: P: 1751161909802845 usec, L: 0
0629 01:51:58.411119 (+ 146us) write_op.cc:270] Start()
0629 01:51:58.411140 (+ 21us) write_op.cc:276] Timestamp: P: 1751161909879300 usec, L: 0
0629 01:51:58.411284 (+ 144us) write_op.cc:270] Start()
0629 01:51:58.411304 (+ 20us) write_op.cc:276] Timestamp: P: 1751161909879859 usec, L: 0
0629 01:51:58.411440 (+ 136us) write_op.cc:270] Start()
0629 01:51:58.411460 (+ 20us) write_op.cc:276] Timestamp: P: 1751161909880365 usec, L: 0
0629 01:51:58.411584 (+ 124us) write_op.cc:270] Start()
0629 01:51:58.411603 (+ 19us) write_op.cc:276] Timestamp: P: 1751161909880938 usec, L: 0
0629 01:51:58.411727 (+ 124us) write_op.cc:270] Start()
0629 01:51:58.411747 (+ 20us) write_op.cc:276] Timestamp: P: 1751161909881418 usec, L: 0
0629 01:51:58.415629 (+ 3882us) write_op.cc:270] Start()
0629 01:51:58.415658 (+ 29us) write_op.cc:276] Timestamp: P: 1751161909881932 usec, L: 0
0629 01:51:58.415814 (+ 156us) write_op.cc:270] Start()
0629 01:51:58.415835 (+ 21us) write_op.cc:276] Timestamp: P: 1751161909882488 usec, L: 0
0629 01:51:58.415964 (+ 129us) write_op.cc:270] Start()
0629 01:51:58.415979 (+ 15us) write_op.cc:276] Timestamp: P: 1751161909883054 usec, L: 0
0629 01:51:58.416190 (+ 211us) write_op.cc:270] Start()
0629 01:51:58.416297 (+ 107us) write_op.cc:276] Timestamp: P: 1751161909896490 usec, L: 0
0629 01:51:58.416492 (+ 195us) write_op.cc:270] Start()
0629 01:51:58.416512 (+ 20us) write_op.cc:276] Timestamp: P: 1751161909904833 usec, L: 0
0629 01:51:58.416634 (+ 122us) write_op.cc:270] Start()
0629 01:51:58.416648 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909993931 usec, L: 0
0629 01:51:58.416751 (+ 103us) write_op.cc:270] Start()
0629 01:51:58.416765 (+ 14us) write_op.cc:276] Timestamp: P: 1751161909994580 usec, L: 0
0629 01:51:58.416868 (+ 103us) write_op.cc:270] Start()
0629 01:51:58.416885 (+ 17us) write_op.cc:276] Timestamp: P: 1751161910038841 usec, L: 0
0629 01:51:58.417013 (+ 128us) write_op.cc:270] Start()
0629 01:51:58.417031 (+ 18us) write_op.cc:276] Timestamp: P: 1751161910041109 usec, L: 0
0629 01:51:58.417165 (+ 134us) write_op.cc:270] Start()
0629 01:51:58.417187 (+ 22us) write_op.cc:276] Timestamp: P: 1751161910043397 usec, L: 0
0629 01:51:58.417340 (+ 153us) write_op.cc:270] Start()
0629 01:51:58.417361 (+ 21us) write_op.cc:276] Timestamp: P: 1751161910081575 usec, L: 0
0629 01:51:58.432743 (+ 15382us) write_op.cc:270] Start()
0629 01:51:58.432768 (+ 25us) write_op.cc:276] Timestamp: P: 1751161910082088 usec, L: 0
0629 01:51:58.432907 (+ 139us) write_op.cc:270] Start()
0629 01:51:58.432928 (+ 21us) write_op.cc:276] Timestamp: P: 1751161910087704 usec, L: 0
0629 01:51:58.433065 (+ 137us) write_op.cc:270] Start()
0629 01:51:58.433083 (+ 18us) write_op.cc:276] Timestamp: P: 1751161910091342 usec, L: 0
0629 01:51:58.433235 (+ 152us) write_op.cc:270] Start()
0629 01:51:58.433252 (+ 17us) write_op.cc:276] Timestamp: P: 1751161910098758 usec, L: 0
0629 01:51:58.433374 (+ 122us) write_op.cc:270] Start()
0629 01:51:58.433391 (+ 17us) write_op.cc:276] Timestamp: P: 1751161910100189 usec, L: 0
0629 01:51:58.433499 (+ 108us) write_op.cc:270] Start()
0629 01:51:58.433514 (+ 15us) write_op.cc:276] Timestamp: P: 1751161910101062 usec, L: 0
0629 01:51:58.433623 (+ 109us) write_op.cc:270] Start()
0629 01:51:58.433638 (+ 15us) write_op.cc:276] Timestamp: P: 1751161910105518 usec, L: 0
0629 01:51:58.433756 (+ 118us) write_op.cc:270] Start()
0629 01:51:58.433772 (+ 16us) write_op.cc:276] Timestamp: P: 1751161910105958 usec, L: 0
0629 01:51:58.433889 (+ 117us) write_op.cc:270] Start()
0629 01:51:58.433905 (+ 16us) write_op.cc:276] Timestamp: P: 1751161910113935 usec, L: 0
0629 01:51:58.434035 (+ 130us) write_op.cc:270] Start()
0629 01:51:58.434053 (+ 18us) write_op.cc:276] Timestamp: P: 1751161910114344 usec, L: 0
0629 01:51:58.434201 (+ 148us) write_op.cc:270] Start()
0629 01:51:58.434221 (+ 20us) write_op.cc:276] Timestamp: P: 1751161910114722 usec, L: 0
0629 01:51:58.434336 (+ 115us) write_op.cc:270] Start()
0629 01:51:58.434352 (+ 16us) write_op.cc:276] Timestamp: P: 1751161910135528 usec, L: 0
0629 01:51:58.434456 (+ 104us) write_op.cc:270] Start()
0629 01:51:58.434471 (+ 15us) write_op.cc:276] Timestamp: P: 1751161910135991 usec, L: 0
0629 01:51:58.434573 (+ 102us) write_op.cc:270] Start()
0629 01:51:58.434589 (+ 16us) write_op.cc:276] Timestamp: P: 1751161910136435 usec, L: 0
0629 01:51:58.434694 (+ 105us) write_op.cc:270] Start()
0629 01:51:58.434711 (+ 17us) write_op.cc:276] Timestamp: P: 1751161910136839 usec, L: 0
0629 01:51:58.449362 (+ 14651us) write_op.cc:270] Start()
0629 01:51:58.449389 (+ 27us) write_op.cc:276] Timestamp: P: 1751161910137234 usec, L: 0
0629 01:51:58.449544 (+ 155us) write_op.cc:270] Start()
0629 01:51:58.449563 (+ 19us) write_op.cc:276] Timestamp: P: 1751161910145107 usec, L: 0
0629 01:51:58.449685 (+ 122us) write_op.cc:270] Start()
0629 01:51:58.449703 (+ 18us) write_op.cc:276] Timestamp: P: 1751161910189005 usec, L: 0
0629 01:51:58.449824 (+ 121us) write_op.cc:270] Start()
0629 01:51:58.449845 (+ 21us) write_op.cc:276] Timestamp: P: 1751161910232401 usec, L: 0
0629 01:51:58.449974 (+ 129us) write_op.cc:270] Start()
0629 01:51:58.450000 (+ 26us) write_op.cc:276] Timestamp: P: 1751161910350521 usec, L: 0
0629 01:51:58.450111 (+ 111us) write_op.cc:270] Start()
0629 01:51:58.450127 (+ 16us) write_op.cc:276] Timestamp: P: 1751161910393010 usec, L: 0
0629 01:51:58.450225 (+ 98us) write_op.cc:270] Start()
0629 01:51:58.450242 (+ 17us) write_op.cc:276] Timestamp: P: 1751161910394456 usec, L: 0
0629 01:51:58.450402 (+ 160us) write_op.cc:270] Start()
0629 01:51:58.450425 (+ 23us) write_op.cc:276] Timestamp: P: 1751161
W20250629 01:52:00.230967 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 332a2808e00c4c52afa3931cb8c53516 consensus state has no leader
I20250629 01:52:00.231220 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_904b8b7a3ab1414a9bd29274648bbb86" } new_table_name: "default.Table_61a4141b84a64b7085c02660cccf1cfd" modify_external_catalogs: true
I20250629 01:52:00.223374 1156 pending_rounds.cc:77] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759222176989184 op_type: NO_OP noop_request { }
I20250629 01:52:00.235620 1156 raft_consensus.cc:2887] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.235808 1156 raft_consensus.cc:2270] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.236650 1156 ts_tablet_manager.cc:1905] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.240258 2580 raft_consensus.cc:491] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.240396 2580 raft_consensus.cc:513] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.240918 2580 leader_election.cc:290] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.241215 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b15300fd8ca7489f9d77f1f8992f8df3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.241351 777 raft_consensus.cc:2466] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:00.241722 986 leader_election.cc:304] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.241925 2580 raft_consensus.cc:2802] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.241990 2580 raft_consensus.cc:491] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.242034 2580 raft_consensus.cc:3058] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.244683 2580 raft_consensus.cc:513] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.245393 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b15300fd8ca7489f9d77f1f8992f8df3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:00.245503 777 raft_consensus.cc:3058] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.246819 2580 leader_election.cc:290] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.247084 1156 ts_tablet_manager.cc:1918] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:00.247820 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b15300fd8ca7489f9d77f1f8992f8df3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:00.252580 988 leader_election.cc:343] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.255787 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1223) took 1124 ms. Trace:
I20250629 01:52:00.255897 1158 rpcz_store.cc:276] 0629 01:51:59.131756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.134790 (+ 3034us) service_pool.cc:225] Handling call
0629 01:52:00.068982 (+934192us) ts_tablet_manager.cc:1145] Deleting tablet 3a81e6f9711c4d398ae09f33a75566ae
0629 01:52:00.068990 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.089243 (+ 20253us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.167208 (+ 77965us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.255774 (+ 88566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933981}
I20250629 01:52:00.256371 1158 tablet_replica.cc:331] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.256570 1158 raft_consensus.cc:2241] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.256745 1158 raft_consensus.cc:2270] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.257222 1158 ts_tablet_manager.cc:1905] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.257292 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b15300fd8ca7489f9d77f1f8992f8df3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:00.258240 988 leader_election.cc:343] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.263252 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94f201340df34d00a02cde31f001d6d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:00.264174 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3181bf94172b4d6f95360a4c1be43602 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.265138 2526 ts_tablet_manager.cc:1918] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.265233 2526 log.cc:1199] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/648b0a0f742b4e7da9eaff6bd8795fc4
I20250629 01:52:00.265564 2526 ts_tablet_manager.cc:1939] T 648b0a0f742b4e7da9eaff6bd8795fc4 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.266283 2526 tablet_replica.cc:331] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:00.266427 2526 raft_consensus.cc:2241] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.266409 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 648b0a0f742b4e7da9eaff6bd8795fc4 (table default.Table_afb98d5207ae4d2d8b97b8577ae3a157 [id=e874833b32de4076921fdf0e2f103a1a]) successfully deleted
I20250629 01:52:00.266541 2526 raft_consensus.cc:2270] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.269032 2637 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:52:00.269268 2637 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:52:00.271921 1154 tablet_bootstrap.cc:492] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.272075 1154 ts_tablet_manager.cc:1397] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.002s
I20250629 01:52:00.272603 2526 ts_tablet_manager.cc:1905] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.273339 1154 raft_consensus.cc:357] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.273542 1154 raft_consensus.cc:383] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.273625 1154 raft_consensus.cc:738] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.273850 1154 consensus_queue.cc:260] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.274307 1154 ts_tablet_manager.cc:1428] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:00.274601 1154 tablet_bootstrap.cc:492] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.276054 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 3a81e6f9711c4d398ae09f33a75566ae (table default.Table_67f686028f0c40c991d3231bd0b0c823 [id=c51c68dd349543d4a4a8bfb1b9aa1184]) successfully deleted
I20250629 01:52:00.247665 777 raft_consensus.cc:2466] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:00.255545 2553 raft_consensus.cc:491] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.279615 2553 raft_consensus.cc:513] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.280798 986 leader_election.cc:304] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:00.281373 2553 leader_election.cc:290] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.282048 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.282177 1051 raft_consensus.cc:2466] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:00.282893 1154 tablet_bootstrap.cc:654] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.282892 710 leader_election.cc:304] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.283120 2534 raft_consensus.cc:2802] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.283181 2534 raft_consensus.cc:491] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.283232 2534 raft_consensus.cc:3058] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.284808 2580 raft_consensus.cc:2802] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.284919 2580 raft_consensus.cc:695] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:00.285090 2580 consensus_queue.cc:237] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.285377 2534 raft_consensus.cc:513] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.285851 2534 leader_election.cc:290] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.286818 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:00.286962 1051 raft_consensus.cc:3058] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.289088 1051 raft_consensus.cc:2466] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:00.290611 1155 tablet_bootstrap.cc:654] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.290987 710 leader_election.cc:304] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.275228 1159 tablet_bootstrap.cc:492] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.292588 1159 ts_tablet_manager.cc:1397] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.138s user 0.017s sys 0.007s
I20250629 01:52:00.281729 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.286500 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:00.293680 712 leader_election.cc:343] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.294312 712 leader_election.cc:343] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.294831 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3fce5fefd74344c29e1895554190fd44" } new_table_name: "default.Table_9ebc7edad3214242b12890d1cc3d81d4" modify_external_catalogs: true
I20250629 01:52:00.294900 1159 raft_consensus.cc:357] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.295024 1159 raft_consensus.cc:383] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.295888 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_238425ea966b4ee1b67de895973a1521"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.296337 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_238425ea966b4ee1b67de895973a1521 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.296358 1158 ts_tablet_manager.cc:1918] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.296438 1158 log.cc:1199] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/819ed44d98f14075b9a67b88dc44e86e
I20250629 01:52:00.296698 1158 ts_tablet_manager.cc:1939] T 819ed44d98f14075b9a67b88dc44e86e P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.297317 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1224) took 1162 ms. Trace:
I20250629 01:52:00.297377 1158 rpcz_store.cc:276] 0629 01:51:59.135137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.136532 (+ 1395us) service_pool.cc:225] Handling call
0629 01:52:00.256330 (+1119798us) ts_tablet_manager.cc:1145] Deleting tablet 819ed44d98f14075b9a67b88dc44e86e
0629 01:52:00.256338 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.259943 (+ 3605us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.296280 (+ 36337us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.297301 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1119657}
I20250629 01:52:00.297696 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 819ed44d98f14075b9a67b88dc44e86e (table default.Table_f6acf9f470204fdb876176dc5c191895 [id=0070a00fcaa945cfb2e47f3c3e347323]) successfully deleted
I20250629 01:52:00.297927 1158 tablet_replica.cc:331] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.298070 1158 raft_consensus.cc:2241] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.298188 1158 raft_consensus.cc:2270] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.298749 892 tablet_service.cc:1468] Processing CreateTablet for tablet b28efcf880c84270affb145cd6f907e4 (DEFAULT_TABLE table=default.Table_36cbb7b4045745cc8a65574c33d2ea65 [id=60e85d7bef3143978b27ce442a5d1714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.299039 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b28efcf880c84270affb145cd6f907e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.299754 742 tablet_service.cc:1468] Processing CreateTablet for tablet b28efcf880c84270affb145cd6f907e4 (DEFAULT_TABLE table=default.Table_36cbb7b4045745cc8a65574c33d2ea65 [id=60e85d7bef3143978b27ce442a5d1714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.299963 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:52:00.300053 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b28efcf880c84270affb145cd6f907e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.300312 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:00.301828 1023 tablet_service.cc:1468] Processing CreateTablet for tablet b28efcf880c84270affb145cd6f907e4 (DEFAULT_TABLE table=default.Table_36cbb7b4045745cc8a65574c33d2ea65 [id=60e85d7bef3143978b27ce442a5d1714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.302105 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b28efcf880c84270affb145cd6f907e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.304543 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c85076a59d9c4336942e15ebb642ed74" } new_table_name: "default.Table_82a5867209d44446a2e3db934706cdaa" modify_external_catalogs: true
I20250629 01:52:00.306545 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e1e22a0c02054c7db514eefe20ee3311 (table default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e]). Expected version 1 got 0
I20250629 01:52:00.306759 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:00.308240 1158 ts_tablet_manager.cc:1905] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.312335 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:52:00.312582 447 catalog_manager.cc:5582] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:00.312877 447 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet cd9a41fae477470281dfba4d4e552b99 (table default.Table_c58b64be9b9040a294ce611a1bd282c7 [id=4525666163d64ba981dfff1c095c6ee2]). Expected version 1 got 0
W20250629 01:52:00.313717 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0daa93d5b22407796027ba8740026e6 consensus state has no leader
I20250629 01:52:00.313769 2553 raft_consensus.cc:2802] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.313875 2553 raft_consensus.cc:695] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:00.314074 2553 consensus_queue.cc:237] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.315006 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 6092e19acf974f2dad5a903da72944ba from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:00.315824 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 24c4841e127b495cbfbca0103662c01e
I20250629 01:52:00.316804 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 24c4841e127b495cbfbca0103662c01e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.316849 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 24c4841e127b495cbfbca0103662c01e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.316883 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 24c4841e127b495cbfbca0103662c01e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.317916 892 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca03c217f2945e9af359b5e5beed555 (DEFAULT_TABLE table=default.Table_9f48c56326b54e74ac850d12a81563cf [id=bd3885186a4b4d418e8554ce2e233c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.318148 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 27b76e04158e492a9979c06e7cc2ae37
I20250629 01:52:00.318212 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca03c217f2945e9af359b5e5beed555. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.318650 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 27b76e04158e492a9979c06e7cc2ae37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.318878 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca03c217f2945e9af359b5e5beed555 (DEFAULT_TABLE table=default.Table_9f48c56326b54e74ac850d12a81563cf [id=bd3885186a4b4d418e8554ce2e233c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.319124 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 27b76e04158e492a9979c06e7cc2ae37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.319166 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca03c217f2945e9af359b5e5beed555. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.320228 742 tablet_service.cc:1468] Processing CreateTablet for tablet 7ca03c217f2945e9af359b5e5beed555 (DEFAULT_TABLE table=default.Table_9f48c56326b54e74ac850d12a81563cf [id=bd3885186a4b4d418e8554ce2e233c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.320541 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ca03c217f2945e9af359b5e5beed555. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.326728 752 tablet_service.cc:1468] Processing CreateTablet for tablet 26c4564df52e49bda8b122312daa8861 (DEFAULT_TABLE table=default.Table_1aba3deb8ac142f9979055dad2418720 [id=c220ceb9c9e24e19a15fc052c7c99fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.327015 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c4564df52e49bda8b122312daa8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.327647 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 26c4564df52e49bda8b122312daa8861 (DEFAULT_TABLE table=default.Table_1aba3deb8ac142f9979055dad2418720 [id=c220ceb9c9e24e19a15fc052c7c99fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.327898 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c4564df52e49bda8b122312daa8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.328025 896 tablet_service.cc:1468] Processing CreateTablet for tablet 26c4564df52e49bda8b122312daa8861 (DEFAULT_TABLE table=default.Table_1aba3deb8ac142f9979055dad2418720 [id=c220ceb9c9e24e19a15fc052c7c99fd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.328318 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26c4564df52e49bda8b122312daa8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.332923 2049 raft_consensus.cc:491] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.333041 2049 raft_consensus.cc:513] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.333482 2049 leader_election.cc:290] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.333571 2049 raft_consensus.cc:491] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.333640 2049 raft_consensus.cc:513] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.334043 2049 leader_election.cc:290] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.334609 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "327c303b6b9347f5b4cb78f74d41a44a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.334627 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a014907d09c94745859436f1eb73b280" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:00.334980 712 leader_election.cc:343] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.335268 712 leader_election.cc:343] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.336158 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "327c303b6b9347f5b4cb78f74d41a44a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.336468 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a014907d09c94745859436f1eb73b280" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.336583 1051 raft_consensus.cc:2466] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:00.337126 880 tablet_service.cc:1468] Processing CreateTablet for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 (DEFAULT_TABLE table=default.Table_226a81ddbfc54450b599ac93e64a0517 [id=99344d8478d0471e9253fdc8e7d2326a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.337428 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7051dc43b84d7bbf70b51f0b9bced8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:00.338253 710 leader_election.cc:343] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.338330 710 leader_election.cc:304] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:00.338619 710 leader_election.cc:304] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:00.338925 2049 consensus_queue.cc:1035] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:00.339210 2049 raft_consensus.cc:2747] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.339331 2049 raft_consensus.cc:2802] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.339372 2049 raft_consensus.cc:491] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.339424 2049 raft_consensus.cc:3058] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.341570 2049 raft_consensus.cc:513] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.342820 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a014907d09c94745859436f1eb73b280" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:00.342970 1052 raft_consensus.cc:3058] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.343914 1155 tablet_bootstrap.cc:492] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.344538 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 27b76e04158e492a9979c06e7cc2ae37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.345052 1052 raft_consensus.cc:2466] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:00.346493 1155 ts_tablet_manager.cc:1397] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.118s user 0.001s sys 0.002s
I20250629 01:52:00.346604 892 tablet_service.cc:1468] Processing CreateTablet for tablet db906e40659f43afac9b8d53c7560897 (DEFAULT_TABLE table=default.Table_d74571bf53c54b24aa34535f695efa89 [id=e2a3e63f32434247bef9da6bacda50c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.347015 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db906e40659f43afac9b8d53c7560897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.347304 1155 raft_consensus.cc:357] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.350139 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a014907d09c94745859436f1eb73b280" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:00.350739 2526 ts_tablet_manager.cc:1918] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.351405 880 tablet_service.cc:1468] Processing CreateTablet for tablet 2fa581a6d6f64f0eb18f976caf675858 (DEFAULT_TABLE table=default.Table_3cd6904b29dc4d52bb6acb613babf904 [id=259db841e6904f4faddfafd0a0e63a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.352442 2526 log.cc:1199] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/d6bf3c7d0bb548e0b77ba8b569a72d0d
I20250629 01:52:00.352694 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fa581a6d6f64f0eb18f976caf675858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.353489 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 (DEFAULT_TABLE table=default.Table_226a81ddbfc54450b599ac93e64a0517 [id=99344d8478d0471e9253fdc8e7d2326a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.353775 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7051dc43b84d7bbf70b51f0b9bced8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.356424 1023 tablet_service.cc:1468] Processing CreateTablet for tablet db906e40659f43afac9b8d53c7560897 (DEFAULT_TABLE table=default.Table_d74571bf53c54b24aa34535f695efa89 [id=e2a3e63f32434247bef9da6bacda50c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.356696 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db906e40659f43afac9b8d53c7560897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.352720 2526 ts_tablet_manager.cc:1939] T d6bf3c7d0bb548e0b77ba8b569a72d0d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.361303 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet d6bf3c7d0bb548e0b77ba8b569a72d0d (table default.Table_f5bc90b9f24640639b11b63857131036 [id=5ebf3881fc6a49b7bf063344e3b5cb40]) successfully deleted
I20250629 01:52:00.361583 2526 tablet_replica.cc:331] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:00.361807 2526 raft_consensus.cc:2241] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.361972 2526 raft_consensus.cc:2270] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.362321 1159 raft_consensus.cc:738] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.362524 1159 consensus_queue.cc:260] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.362915 1159 ts_tablet_manager.cc:1428] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.070s user 0.000s sys 0.000s
I20250629 01:52:00.365511 2625 raft_consensus.cc:491] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.365617 2625 raft_consensus.cc:513] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.366086 2625 leader_election.cc:290] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:00.366431 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d456410cd09845dabd979c123f70741b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.367240 1159 tablet_bootstrap.cc:492] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.368921 1159 tablet_bootstrap.cc:654] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.362601 2526 ts_tablet_manager.cc:1905] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.377094 1154 tablet_bootstrap.cc:492] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.377578 1154 ts_tablet_manager.cc:1397] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.103s user 0.014s sys 0.004s
I20250629 01:52:00.378351 1154 raft_consensus.cc:357] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.349519 710 leader_election.cc:304] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:52:00.378829 712 leader_election.cc:343] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.378917 2548 raft_consensus.cc:2802] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.379527 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d456410cd09845dabd979c123f70741b" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.380024 916 raft_consensus.cc:3058] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.350841 2049 leader_election.cc:290] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.351480 2534 raft_consensus.cc:491] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.384512 2534 raft_consensus.cc:513] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.385103 2534 leader_election.cc:290] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.355796 752 tablet_service.cc:1468] Processing CreateTablet for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 (DEFAULT_TABLE table=default.Table_226a81ddbfc54450b599ac93e64a0517 [id=99344d8478d0471e9253fdc8e7d2326a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.385838 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f7051dc43b84d7bbf70b51f0b9bced8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.357458 2553 raft_consensus.cc:491] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.358028 750 tablet_service.cc:1468] Processing CreateTablet for tablet db906e40659f43afac9b8d53c7560897 (DEFAULT_TABLE table=default.Table_d74571bf53c54b24aa34535f695efa89 [id=e2a3e63f32434247bef9da6bacda50c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.387781 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db906e40659f43afac9b8d53c7560897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.392007 1155 raft_consensus.cc:383] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.358745 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2fa581a6d6f64f0eb18f976caf675858 (DEFAULT_TABLE table=default.Table_3cd6904b29dc4d52bb6acb613babf904 [id=259db841e6904f4faddfafd0a0e63a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.392066 1155 raft_consensus.cc:738] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.392282 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fa581a6d6f64f0eb18f976caf675858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.392231 1155 consensus_queue.cc:260] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.392639 1155 ts_tablet_manager.cc:1428] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.046s user 0.001s sys 0.000s
I20250629 01:52:00.393380 1155 tablet_bootstrap.cc:492] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.394481 1155 tablet_bootstrap.cc:654] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.395434 2652 raft_consensus.cc:491] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.395553 2652 raft_consensus.cc:513] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.396019 2652 leader_election.cc:290] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:00.396421 2553 raft_consensus.cc:513] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.396468 1154 raft_consensus.cc:383] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.396539 1154 raft_consensus.cc:738] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.388569 1158 ts_tablet_manager.cc:1918] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.396790 1158 log.cc:1199] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f4b7d0a44bbe4d8c81e8d0c444dbb55b
I20250629 01:52:00.396767 1154 consensus_queue.cc:260] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.396888 2553 leader_election.cc:290] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.397094 1158 ts_tablet_manager.cc:1939] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.397594 1154 ts_tablet_manager.cc:1428] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.020s user 0.000s sys 0.001s
I20250629 01:52:00.398205 757 tablet_service.cc:1468] Processing CreateTablet for tablet f4cdefd680414cbda42e4090e78b1fd2 (DEFAULT_TABLE table=default.Table_dae51ebf34aa422a9e59b9369d5564a0 [id=f064321537ad49aeab2f63a840b16764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.398346 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1225) took 1261 ms. Trace:
I20250629 01:52:00.398488 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cdefd680414cbda42e4090e78b1fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.398409 1158 rpcz_store.cc:276] 0629 01:51:59.136820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.138571 (+ 1751us) service_pool.cc:225] Handling call
0629 01:52:00.297891 (+1159320us) ts_tablet_manager.cc:1145] Deleting tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b
0629 01:52:00.297899 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.314081 (+ 16182us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.388475 (+ 74394us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.398331 (+ 9856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1159204}
I20250629 01:52:00.398816 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b (table default.Table_2acf6246904d41ab94aad9dec7df2291 [id=df90a81df72e4bc09675d9b0203888fc]) successfully deleted
I20250629 01:52:00.399111 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1240) took 1184 ms. Trace:
I20250629 01:52:00.399178 1158 rpcz_store.cc:276] 0629 01:51:59.214743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.214789 (+ 46us) service_pool.cc:225] Handling call
0629 01:52:00.399015 (+1184226us) ts_tablet_manager.cc:1145] Deleting tablet 1a5d7cc320ca455e87d16a78c5f0407d
0629 01:52:00.399022 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.399098 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1122540}
W20250629 01:52:00.399485 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1a5d7cc320ca455e87d16a78c5f0407d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1a5d7cc320ca455e87d16a78c5f0407d already in progress: creating tablet
I20250629 01:52:00.399781 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1241) took 1184 ms. Trace:
I20250629 01:52:00.399860 1154 tablet_bootstrap.cc:492] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.399839 1158 rpcz_store.cc:276] 0629 01:51:59.214845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.214884 (+ 39us) service_pool.cc:225] Handling call
0629 01:52:00.399676 (+1184792us) ts_tablet_manager.cc:1145] Deleting tablet 6d7afa4be82a4ae3974bd0b4786d22db
0629 01:52:00.399684 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.399768 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123153}
W20250629 01:52:00.400426 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6d7afa4be82a4ae3974bd0b4786d22db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d7afa4be82a4ae3974bd0b4786d22db already in progress: creating tablet
I20250629 01:52:00.400691 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1242) took 1185 ms. Trace:
I20250629 01:52:00.400745 1158 rpcz_store.cc:276] 0629 01:51:59.215062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215104 (+ 42us) service_pool.cc:225] Handling call
0629 01:52:00.400607 (+1185503us) ts_tablet_manager.cc:1145] Deleting tablet 84cfa2d73c6b4ca2bf41437b196d10b1
0629 01:52:00.400614 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.400677 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124050}
I20250629 01:52:00.400933 1154 tablet_bootstrap.cc:654] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:00.401057 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 84cfa2d73c6b4ca2bf41437b196d10b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84cfa2d73c6b4ca2bf41437b196d10b1 already in progress: creating tablet
I20250629 01:52:00.401484 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1243) took 1186 ms. Trace:
I20250629 01:52:00.401536 1021 tablet_service.cc:1468] Processing CreateTablet for tablet d4b4e75281ba4cd8a8e1f0a47710964f (DEFAULT_TABLE table=default.Table_a1e8b8763bff46768cb4e15ba61d470b [id=9d8c7b07069d4b86b2a5e18fa3e88ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.402019 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b4e75281ba4cd8a8e1f0a47710964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.404367 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2fa581a6d6f64f0eb18f976caf675858 (DEFAULT_TABLE table=default.Table_3cd6904b29dc4d52bb6acb613babf904 [id=259db841e6904f4faddfafd0a0e63a89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.404800 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fa581a6d6f64f0eb18f976caf675858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.405717 880 tablet_service.cc:1468] Processing CreateTablet for tablet d4b4e75281ba4cd8a8e1f0a47710964f (DEFAULT_TABLE table=default.Table_a1e8b8763bff46768cb4e15ba61d470b [id=9d8c7b07069d4b86b2a5e18fa3e88ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.406018 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b4e75281ba4cd8a8e1f0a47710964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.411712 892 tablet_service.cc:1468] Processing CreateTablet for tablet f4cdefd680414cbda42e4090e78b1fd2 (DEFAULT_TABLE table=default.Table_dae51ebf34aa422a9e59b9369d5564a0 [id=f064321537ad49aeab2f63a840b16764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.412391 896 tablet_service.cc:1468] Processing CreateTablet for tablet 52a90e9c1e804a67b82cb01c105af698 (DEFAULT_TABLE table=default.Table_2b15bee7396749e4bf96b95d85662564 [id=2a5e97bf190d4a2a90c90bb92b5eb1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.413256 1016 tablet_service.cc:1468] Processing CreateTablet for tablet f4cdefd680414cbda42e4090e78b1fd2 (DEFAULT_TABLE table=default.Table_dae51ebf34aa422a9e59b9369d5564a0 [id=f064321537ad49aeab2f63a840b16764]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.401563 1158 rpcz_store.cc:276] 0629 01:51:59.215281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215321 (+ 40us) service_pool.cc:225] Handling call
0629 01:52:00.401372 (+1186051us) ts_tablet_manager.cc:1145] Deleting tablet cc65674750c64114b999e0e292a851f9
0629 01:52:00.401380 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.401465 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1124781}
I20250629 01:52:00.413532 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cdefd680414cbda42e4090e78b1fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.413609 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1244) took 1198 ms. Trace:
I20250629 01:52:00.413669 1158 rpcz_store.cc:276] 0629 01:51:59.215357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215392 (+ 35us) service_pool.cc:225] Handling call
0629 01:52:00.413517 (+1198125us) ts_tablet_manager.cc:1145] Deleting tablet 1e87caeaa536423685a228252bb9a901
0629 01:52:00.413525 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.413595 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1136886}
I20250629 01:52:00.413705 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4c34a9c407304a6e825c61d81bde6ac2 (DEFAULT_TABLE table=default.Table_3f36a3b189db4f4296fd81d47125b49c [id=d536fd8b154449759b12f53a6ec7970f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.413867 1158 tablet_replica.cc:331] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.414016 1158 raft_consensus.cc:2241] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.414130 1158 raft_consensus.cc:2270] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.414252 885 tablet_service.cc:1468] Processing CreateTablet for tablet 4e2531956f6a48e1a94717eb3cf35cd9 (DEFAULT_TABLE table=default.Table_4658a2e500d648bc9e25fd35a86d523f [id=2d72b4de4c5243e69834027a0384687a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.414552 1158 ts_tablet_manager.cc:1905] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.415161 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4c34a9c407304a6e825c61d81bde6ac2 (DEFAULT_TABLE table=default.Table_3f36a3b189db4f4296fd81d47125b49c [id=d536fd8b154449759b12f53a6ec7970f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.415418 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c34a9c407304a6e825c61d81bde6ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.413532 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 52a90e9c1e804a67b82cb01c105af698 (DEFAULT_TABLE table=default.Table_2b15bee7396749e4bf96b95d85662564 [id=2a5e97bf190d4a2a90c90bb92b5eb1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.416811 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a90e9c1e804a67b82cb01c105af698. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.418728 742 tablet_service.cc:1468] Processing CreateTablet for tablet d4b4e75281ba4cd8a8e1f0a47710964f (DEFAULT_TABLE table=default.Table_a1e8b8763bff46768cb4e15ba61d470b [id=9d8c7b07069d4b86b2a5e18fa3e88ddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.418977 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4b4e75281ba4cd8a8e1f0a47710964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.420049 877 tablet_service.cc:1468] Processing CreateTablet for tablet 57eb4e64e9e547778f1a6acf37e6f251 (DEFAULT_TABLE table=default.Table_910f8f6757c44909a04eb629bfa037e8 [id=5daea2259d3c424eae96f924ca9440da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.420452 884 tablet_service.cc:1468] Processing CreateTablet for tablet 5b8f93f0bbe14ccfa00fa92179535d49 (DEFAULT_TABLE table=default.Table_ed0a310f4015427690bfcc25f7618ada [id=f7b9c9d3dbf948eead6a3a71a50beed7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:00.421561 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet cc65674750c64114b999e0e292a851f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc65674750c64114b999e0e292a851f9 already in progress: creating tablet
W20250629 01:52:00.422035 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1e87caeaa536423685a228252bb9a901 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e87caeaa536423685a228252bb9a901 already in progress: creating tablet
I20250629 01:52:00.422240 2558 tablet.cc:1722] T e5e8b8e4d42f43c78f34102b76b4b11a P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.424002 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 57eb4e64e9e547778f1a6acf37e6f251 (DEFAULT_TABLE table=default.Table_910f8f6757c44909a04eb629bfa037e8 [id=5daea2259d3c424eae96f924ca9440da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.424304 757 tablet_service.cc:1468] Processing CreateTablet for tablet 52a90e9c1e804a67b82cb01c105af698 (DEFAULT_TABLE table=default.Table_2b15bee7396749e4bf96b95d85662564 [id=2a5e97bf190d4a2a90c90bb92b5eb1bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.424571 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a90e9c1e804a67b82cb01c105af698. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.424645 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57eb4e64e9e547778f1a6acf37e6f251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.425974 746 tablet_service.cc:1468] Processing CreateTablet for tablet 57eb4e64e9e547778f1a6acf37e6f251 (DEFAULT_TABLE table=default.Table_910f8f6757c44909a04eb629bfa037e8 [id=5daea2259d3c424eae96f924ca9440da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.426241 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57eb4e64e9e547778f1a6acf37e6f251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.428149 2652 raft_consensus.cc:491] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.428287 2652 raft_consensus.cc:513] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.428830 2652 leader_election.cc:290] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.429499 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4cdefd680414cbda42e4090e78b1fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.429787 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2531956f6a48e1a94717eb3cf35cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.432214 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57eb4e64e9e547778f1a6acf37e6f251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.436568 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8f93f0bbe14ccfa00fa92179535d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.438199 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8295e8178eff4d5eb95ac885092731e0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.439342 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce90c045115844b0b96143f91f1be1d4" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.439476 1048 raft_consensus.cc:2466] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:00.440405 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c34a9c407304a6e825c61d81bde6ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.440797 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f94d8fc1b6254ba69acadcaff3613d97" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.440917 1048 raft_consensus.cc:2466] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:00.441725 2578 tablet.cc:1722] T e5e8b8e4d42f43c78f34102b76b4b11a P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.444283 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52a90e9c1e804a67b82cb01c105af698. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:00.447548 710 leader_election.cc:343] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.447719 710 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: fca257d6cb614110948e1045dee44bdc
I20250629 01:52:00.447840 710 leader_election.cc:304] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.448019 2049 raft_consensus.cc:2802] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.448081 2049 raft_consensus.cc:491] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.448163 2049 raft_consensus.cc:3058] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.449275 1159 tablet_bootstrap.cc:492] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.449384 1159 ts_tablet_manager.cc:1397] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.000s
I20250629 01:52:00.450078 1159 raft_consensus.cc:357] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.450217 1159 raft_consensus.cc:383] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.450280 1159 raft_consensus.cc:738] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.450351 2049 raft_consensus.cc:513] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.450515 1159 consensus_queue.cc:260] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.450902 2049 leader_election.cc:290] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.451007 1159 ts_tablet_manager.cc:1428] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:00.451223 1159 tablet_bootstrap.cc:492] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.451804 1155 tablet_bootstrap.cc:492] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.451987 1155 ts_tablet_manager.cc:1397] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.000s
I20250629 01:52:00.452785 1155 raft_consensus.cc:357] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.452960 1155 raft_consensus.cc:383] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.453050 1155 raft_consensus.cc:738] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.453105 1159 tablet_bootstrap.cc:654] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.453281 1155 consensus_queue.cc:260] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.453755 1155 ts_tablet_manager.cc:1428] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:00.454062 1155 tablet_bootstrap.cc:492] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.455116 1155 tablet_bootstrap.cc:654] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.456652 2526 ts_tablet_manager.cc:1918] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.456776 2526 log.cc:1199] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e3b17172ab8b4e0dad36ebaac776dc03
I20250629 01:52:00.457038 2526 ts_tablet_manager.cc:1939] T e3b17172ab8b4e0dad36ebaac776dc03 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.457794 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e3b17172ab8b4e0dad36ebaac776dc03 (table default.Table_11a668da1f204d02b70b4ccd24b59bac [id=4fef3acd201d4a7c830ec534cdee8e3f]) successfully deleted
I20250629 01:52:00.455544 710 leader_election.cc:304] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.459609 2649 raft_consensus.cc:2802] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.459730 2649 raft_consensus.cc:491] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.459836 2649 raft_consensus.cc:3058] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.460443 2548 raft_consensus.cc:695] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:00.460693 2548 consensus_queue.cc:237] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:00.461560 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 82139.1us
I20250629 01:52:00.461833 850 leader_election.cc:304] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.463053 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce90c045115844b0b96143f91f1be1d4" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:00.463193 1048 raft_consensus.cc:3058] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.463476 2625 raft_consensus.cc:2747] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.465407 1048 raft_consensus.cc:2466] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:00.465592 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4e2531956f6a48e1a94717eb3cf35cd9 (DEFAULT_TABLE table=default.Table_4658a2e500d648bc9e25fd35a86d523f [id=2d72b4de4c5243e69834027a0384687a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.465983 710 leader_election.cc:304] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.466042 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2531956f6a48e1a94717eb3cf35cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.466409 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8295e8178eff4d5eb95ac885092731e0" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.466519 2534 raft_consensus.cc:2802] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.466621 2534 raft_consensus.cc:695] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:00.466809 2534 consensus_queue.cc:237] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.468405 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4e2531956f6a48e1a94717eb3cf35cd9 (DEFAULT_TABLE table=default.Table_4658a2e500d648bc9e25fd35a86d523f [id=2d72b4de4c5243e69834027a0384687a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.468837 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2531956f6a48e1a94717eb3cf35cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.469904 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_81bfc89e69ed457089280d6e2049316e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.470423 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81bfc89e69ed457089280d6e2049316e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.471447 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_704f6c68c7484e5582a42cef6e942588"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.471822 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_704f6c68c7484e5582a42cef6e942588 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.472954 912 raft_consensus.cc:3058] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.473449 750 tablet_service.cc:1468] Processing CreateTablet for tablet 4c34a9c407304a6e825c61d81bde6ac2 (DEFAULT_TABLE table=default.Table_3f36a3b189db4f4296fd81d47125b49c [id=d536fd8b154449759b12f53a6ec7970f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.473779 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c34a9c407304a6e825c61d81bde6ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.474545 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_830b38557e8d4844ae1ab213c459309b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.474901 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_830b38557e8d4844ae1ab213c459309b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.475153 752 tablet_service.cc:1468] Processing CreateTablet for tablet 5b8f93f0bbe14ccfa00fa92179535d49 (DEFAULT_TABLE table=default.Table_ed0a310f4015427690bfcc25f7618ada [id=f7b9c9d3dbf948eead6a3a71a50beed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.475759 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8f93f0bbe14ccfa00fa92179535d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.476737 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce90c045115844b0b96143f91f1be1d4" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.477085 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 5b8f93f0bbe14ccfa00fa92179535d49 (DEFAULT_TABLE table=default.Table_ed0a310f4015427690bfcc25f7618ada [id=f7b9c9d3dbf948eead6a3a71a50beed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:00.477496 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b8f93f0bbe14ccfa00fa92179535d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:00.483520 2649 raft_consensus.cc:513] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.484789 2626 raft_consensus.cc:491] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.484889 2626 raft_consensus.cc:513] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.485352 2626 leader_election.cc:290] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:00.485849 2649 leader_election.cc:290] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.485927 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.486284 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f94d8fc1b6254ba69acadcaff3613d97" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:00.486411 1051 raft_consensus.cc:3058] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.487143 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce90c045115844b0b96143f91f1be1d4" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:00.488586 1051 raft_consensus.cc:2466] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:00.488797 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f94d8fc1b6254ba69acadcaff3613d97" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.490196 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4b11d32f1c3e41f3b9f6d20fd837226a
I20250629 01:52:00.490772 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5ef79ed619d54eb28e78246a1a4f2b0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:00.490975 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11d32f1c3e41f3b9f6d20fd837226a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:00.491101 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ef79ed619d54eb28e78246a1a4f2b0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.491967 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9c66df5169224fea8c4fc45df7d695d8
I20250629 01:52:00.492537 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8ccb29231d6746ab93412d34b939117f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.492856 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ccb29231d6746ab93412d34b939117f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.494274 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11d32f1c3e41f3b9f6d20fd837226a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.494875 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c66df5169224fea8c4fc45df7d695d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.500743 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c66df5169224fea8c4fc45df7d695d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.501092 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet f93f07e50b414db29c081760ce2004c1
I20250629 01:52:00.501636 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet f93f07e50b414db29c081760ce2004c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:00.502017 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf676660c9b44ffd87e365287e6fc0a9 consensus state has no leader
I20250629 01:52:00.502202 449 master_service.cc:948] ReplaceTablet: received request to replace tablet b5bcfb2ac1674beb8e9d50d2818cec0d from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:00.508448 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:52:00.509150 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec] alter complete (version 1)
I20250629 01:52:00.510314 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e033aedd05ca4aef91836f468c2adfb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.510716 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e033aedd05ca4aef91836f468c2adfb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.511361 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ae6c196463434b00a4de0e1349115428"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.511718 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6c196463434b00a4de0e1349115428 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.501643 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b4abe11363964e61a884d8bf8f8d0189" } new_table_name: "default.Table_e2f8e877756d4bf1a9535269e5e28bfc" modify_external_catalogs: true
I20250629 01:52:00.513411 892 tablet_service.cc:1515] Processing DeleteTablet for tablet f93f07e50b414db29c081760ce2004c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.514979 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_fafd19402c4f4949ba4b7a4bf44473aa" } new_table_name: "default.Table_e2356e15845c43e6b8b82db4ce049d93" modify_external_catalogs: true
W20250629 01:52:00.487663 712 leader_election.cc:343] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Not found: Tablet not found: fca257d6cb614110948e1045dee44bdc
I20250629 01:52:00.517840 712 leader_election.cc:304] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:00.494563 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 4b11d32f1c3e41f3b9f6d20fd837226a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.495493 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 9c66df5169224fea8c4fc45df7d695d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.520613 2651 raft_consensus.cc:2747] T fca257d6cb614110948e1045dee44bdc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:52:00.521275 1154 tablet_bootstrap.cc:492] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.521441 1154 ts_tablet_manager.cc:1397] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.122s user 0.035s sys 0.021s
W20250629 01:52:00.521687 712 leader_election.cc:343] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.521976 712 leader_election.cc:304] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:00.522017 1154 raft_consensus.cc:357] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.522147 1154 raft_consensus.cc:383] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.522218 1154 raft_consensus.cc:738] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.522442 1154 consensus_queue.cc:260] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.522495 2651 raft_consensus.cc:2747] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.503315 750 tablet_service.cc:1515] Processing DeleteTablet for tablet f93f07e50b414db29c081760ce2004c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.509904 757 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.510003 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.510643 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b0782c7e652419096710563930f2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.510748 753 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.510836 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.512270 2534 raft_consensus.cc:491] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.524039 2534 raft_consensus.cc:513] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.523566 2651 consensus_queue.cc:1035] T 2ad265c77d0047c38430242c53182e01 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250629 01:52:00.524597 2534 leader_election.cc:290] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.512450 710 leader_election.cc:304] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.524811 1154 ts_tablet_manager.cc:1428] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:52:00.525080 1154 tablet_bootstrap.cc:492] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.525220 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "303f77a729c648f9b1146d11056b242d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.525759 2649 raft_consensus.cc:2802] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.526332 1154 tablet_bootstrap.cc:654] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.526327 2649 raft_consensus.cc:695] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
W20250629 01:52:00.526609 710 leader_election.cc:343] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.526993 2649 consensus_queue.cc:237] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.536569 1158 ts_tablet_manager.cc:1918] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.536819 1158 log.cc:1199] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f3bdc8bda8a940be865679a05b90bd08
I20250629 01:52:00.537775 1158 ts_tablet_manager.cc:1939] T f3bdc8bda8a940be865679a05b90bd08 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.538637 2526 tablet_replica.cc:331] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:00.539202 2526 raft_consensus.cc:2241] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.539664 2526 raft_consensus.cc:2270] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.541738 2526 ts_tablet_manager.cc:1905] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:00.544297 712 leader_election.cc:343] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.545231 712 consensus_peers.cc:489] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:00.547636 712 leader_election.cc:343] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.550567 712 leader_election.cc:343] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.555006 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f94d8fc1b6254ba69acadcaff3613d97" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:00.556790 1159 tablet_bootstrap.cc:492] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.556932 1159 ts_tablet_manager.cc:1397] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.106s user 0.000s sys 0.003s
I20250629 01:52:00.557567 1159 raft_consensus.cc:357] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.560209 1159 raft_consensus.cc:383] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.560300 1159 raft_consensus.cc:738] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.560510 1159 consensus_queue.cc:260] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.560932 2487 tablet.cc:1722] T e5e8b8e4d42f43c78f34102b76b4b11a P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.562492 1159 ts_tablet_manager.cc:1428] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:52:00.562768 1159 tablet_bootstrap.cc:492] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.573504 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 1bc3d470df0d4814a655bcff68c956e2 (table default.Table_62e935463ed04d11b4774769be58c37d [id=c2d2657ad07e4e79b27a0ae6b2d6dbec]). Expected version 1 got 0
I20250629 01:52:00.573678 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet d456410cd09845dabd979c123f70741b (table default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70]). Expected version 3 got 0
I20250629 01:52:00.574020 1154 tablet_bootstrap.cc:492] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.574127 1154 ts_tablet_manager.cc:1397] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.005s
I20250629 01:52:00.574683 1154 raft_consensus.cc:357] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.574815 1154 raft_consensus.cc:383] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.574859 1154 raft_consensus.cc:738] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.575016 1154 consensus_queue.cc:260] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.575416 1154 ts_tablet_manager.cc:1428] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:00.575650 1154 tablet_bootstrap.cc:492] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.576753 1154 tablet_bootstrap.cc:654] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.577989 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 8a880c9bb4b94d3b9b12f28d48565f06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.579620 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "303f77a729c648f9b1146d11056b242d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.576027 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet cd9a41fae477470281dfba4d4e552b99 (table default.Table_c58b64be9b9040a294ce611a1bd282c7 [id=4525666163d64ba981dfff1c095c6ee2]). Expected version 1 got 0
I20250629 01:52:00.580277 452 catalog_manager.cc:5582] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:00.580674 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:00.576855 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0a2224d72ed24f8f938582dfde5398dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.581669 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a2224d72ed24f8f938582dfde5398dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.582554 1155 tablet_bootstrap.cc:492] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.582654 1155 ts_tablet_manager.cc:1397] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.129s user 0.002s sys 0.000s
I20250629 01:52:00.577287 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_90ea22302d294dd48870c71a89b49025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.582952 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90ea22302d294dd48870c71a89b49025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.583278 1155 raft_consensus.cc:357] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.583415 1155 raft_consensus.cc:383] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.583460 1155 raft_consensus.cc:738] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.583611 1155 consensus_queue.cc:260] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.578277 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_985b5a9e7b454754a12e4e3fdb38a3a4" } new_table_name: "default.Table_9c37a49e7730465f9c278760fba0c5dd" modify_external_catalogs: true
I20250629 01:52:00.584020 1155 ts_tablet_manager.cc:1428] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:52:00.584602 2548 raft_consensus.cc:491] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.584733 2548 raft_consensus.cc:513] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.585211 2548 leader_election.cc:290] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.585587 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bdc6a26b3594d6fb33a27cac5b29f4f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:00.586061 710 leader_election.cc:343] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.586205 1155 tablet_bootstrap.cc:492] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.587281 1155 tablet_bootstrap.cc:654] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.590831 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b5bcfb2ac1674beb8e9d50d2818cec0d
I20250629 01:52:00.591199 449 catalog_manager.cc:6652] ReplaceTablet: tablet b5bcfb2ac1674beb8e9d50d2818cec0d deleted and replaced by tablet 25a6407044644188922f727d2a01de8e
I20250629 01:52:00.591964 892 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bcfb2ac1674beb8e9d50d2818cec0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25a6407044644188922f727d2a01de8e) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.593014 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bcfb2ac1674beb8e9d50d2818cec0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25a6407044644188922f727d2a01de8e) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.594394 752 tablet_service.cc:1515] Processing DeleteTablet for tablet b5bcfb2ac1674beb8e9d50d2818cec0d with delete_type TABLET_DATA_DELETED (replaced by tablet 25a6407044644188922f727d2a01de8e) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.597473 2626 raft_consensus.cc:491] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.597584 2626 raft_consensus.cc:513] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.598078 2626 leader_election.cc:290] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:00.598690 850 leader_election.cc:343] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 56b60431b72e4792aa1c32fdee40cdb9
I20250629 01:52:00.600378 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1245) took 1384 ms. Trace:
I20250629 01:52:00.600451 1158 rpcz_store.cc:276] 0629 01:51:59.215572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215613 (+ 41us) service_pool.cc:225] Handling call
0629 01:52:00.413832 (+1198219us) ts_tablet_manager.cc:1145] Deleting tablet f3bdc8bda8a940be865679a05b90bd08
0629 01:52:00.413838 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.470172 (+ 56334us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.536365 (+ 66193us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.600363 (+ 63998us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1137167}
I20250629 01:52:00.600772 1158 tablet_replica.cc:331] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.600930 1158 raft_consensus.cc:2241] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:00.601102 1158 pending_rounds.cc:70] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:00.601152 1158 pending_rounds.cc:77] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759224049274880 op_type: NO_OP noop_request { }
I20250629 01:52:00.601224 1158 raft_consensus.cc:2887] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.601286 1158 raft_consensus.cc:2270] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.601688 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f3bdc8bda8a940be865679a05b90bd08 (table default.Table_44f5d1913b0e4eedaac117acfdf02c22 [id=b6e2a7a8ccc245899af18bfd45e0f740]) successfully deleted
I20250629 01:52:00.604475 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c58b64be9b9040a294ce611a1bd282c7" } new_table_name: "default.Table_e7206ec77eb44e729b77a7bda6dea373" modify_external_catalogs: true
I20250629 01:52:00.608374 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ad265c77d0047c38430242c53182e01" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.609485 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3bdc6a26b3594d6fb33a27cac5b29f4f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.611533 850 leader_election.cc:304] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:00.612123 850 leader_election.cc:343] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 56b60431b72e4792aa1c32fdee40cdb9
I20250629 01:52:00.612211 850 leader_election.cc:304] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.612526 2625 raft_consensus.cc:2747] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.612936 1159 tablet_bootstrap.cc:654] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:00.612923 712 leader_election.cc:343] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.613835 712 leader_election.cc:343] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.613919 712 leader_election.cc:304] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:00.614081 2651 raft_consensus.cc:2747] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.614748 1158 ts_tablet_manager.cc:1905] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:00.616429 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 697e056a55764ed9846cb02442579d03 consensus state has no leader
W20250629 01:52:00.616645 712 leader_election.cc:343] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.617067 712 leader_election.cc:304] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:00.617456 2649 raft_consensus.cc:2747] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.621274 2526 ts_tablet_manager.cc:1918] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.621407 2526 log.cc:1199] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0ba9eabd06174b87a1485f4921895c5f
I20250629 01:52:00.621681 2526 ts_tablet_manager.cc:1939] T 0ba9eabd06174b87a1485f4921895c5f P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.622239 1158 ts_tablet_manager.cc:1918] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:00.622310 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1e4a069a8f0a46d0a3f9da0bb78914be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.622738 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e4a069a8f0a46d0a3f9da0bb78914be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.627341 2531 catalog_manager.cc:1261] Loaded cluster ID: 3bd8135387974b63b91ae240fae34427
I20250629 01:52:00.628049 1156 log.cc:1199] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/819ed44d98f14075b9a67b88dc44e86e
I20250629 01:52:00.628239 2531 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: loading cluster ID for follower catalog manager: success
I20250629 01:52:00.628625 1156 ts_tablet_manager.cc:1939] T 819ed44d98f14075b9a67b88dc44e86e P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.624730 1154 tablet_bootstrap.cc:492] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.634928 1158 log.cc:1199] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/9fe9674ffeb44ba6ab60de59d4c08670
I20250629 01:52:00.634975 1154 ts_tablet_manager.cc:1397] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.004s
I20250629 01:52:00.635496 1154 raft_consensus.cc:357] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.635632 1154 raft_consensus.cc:383] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.635668 1154 raft_consensus.cc:738] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.635655 2649 raft_consensus.cc:491] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.635774 2649 raft_consensus.cc:513] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.635818 1154 consensus_queue.cc:260] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.636232 1154 ts_tablet_manager.cc:1428] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:00.636320 2649 leader_election.cc:290] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.636716 1154 tablet_bootstrap.cc:492] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.636756 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a9d2aa33a5c4d9ebecaa83c32b39dcb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:00.637156 710 leader_election.cc:343] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.637907 1158 ts_tablet_manager.cc:1939] T 9fe9674ffeb44ba6ab60de59d4c08670 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.637982 1154 tablet_bootstrap.cc:654] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.640239 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1246) took 1424 ms. Trace:
I20250629 01:52:00.640499 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 819ed44d98f14075b9a67b88dc44e86e (table default.Table_f6acf9f470204fdb876176dc5c191895 [id=0070a00fcaa945cfb2e47f3c3e347323]) successfully deleted
I20250629 01:52:00.646450 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_441db41f85564f6ba3c4756f8d4f65a3" } modify_external_catalogs: true
I20250629 01:52:00.646571 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_441db41f85564f6ba3c4756f8d4f65a3" } modify_external_catalogs: true
I20250629 01:52:00.653654 1158 rpcz_store.cc:276] 0629 01:51:59.215648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215700 (+ 52us) service_pool.cc:225] Handling call
0629 01:52:00.600735 (+1385035us) ts_tablet_manager.cc:1145] Deleting tablet 9fe9674ffeb44ba6ab60de59d4c08670
0629 01:52:00.600742 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.619679 (+ 18937us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.622165 (+ 2486us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.640224 (+ 18059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1324010}
I20250629 01:52:00.654143 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 9fe9674ffeb44ba6ab60de59d4c08670 (table default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]) successfully deleted
W20250629 01:52:00.654471 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 504c15b3b168499d9a82584f82a0b792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 504c15b3b168499d9a82584f82a0b792 already in progress: creating tablet
I20250629 01:52:00.654564 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1247) took 1438 ms. Trace:
I20250629 01:52:00.654630 1158 rpcz_store.cc:276] 0629 01:51:59.215737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215848 (+ 111us) service_pool.cc:225] Handling call
0629 01:52:00.654461 (+1438613us) ts_tablet_manager.cc:1145] Deleting tablet b5c32f8431be4eda8c77641c20ad4d1e
0629 01:52:00.654469 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.654550 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377690}
W20250629 01:52:00.654820 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8ae0d61d254c4e1ba508e30aa35600ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae0d61d254c4e1ba508e30aa35600ca already in progress: creating tablet
W20250629 01:52:00.655076 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 77f6bab8deb740cc9cf43832196e404c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f6bab8deb740cc9cf43832196e404c already in progress: creating tablet
W20250629 01:52:00.655385 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b5c32f8431be4eda8c77641c20ad4d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5c32f8431be4eda8c77641c20ad4d1e already in progress: creating tablet
W20250629 01:52:00.655596 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c969377aaf034e9b896c1b53366555ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c969377aaf034e9b896c1b53366555ff already in progress: creating tablet
I20250629 01:52:00.655855 1158 tablet_replica.cc:331] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
W20250629 01:52:00.655845 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet bbe1f8083a884cdd97afce83bd22073c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe1f8083a884cdd97afce83bd22073c already in progress: creating tablet
I20250629 01:52:00.656016 1158 raft_consensus.cc:2241] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:00.656219 1158 pending_rounds.cc:70] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:00.656270 1158 pending_rounds.cc:77] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759223786397696 op_type: NO_OP noop_request { }
I20250629 01:52:00.656347 1158 raft_consensus.cc:2887] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.656409 1158 raft_consensus.cc:2270] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.656586 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3dcc0af85f1241a1a768b66f237b4087" } modify_external_catalogs: true
I20250629 01:52:00.656694 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3dcc0af85f1241a1a768b66f237b4087" } modify_external_catalogs: true
I20250629 01:52:00.658140 1156 tablet_replica.cc:331] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:00.658310 1156 raft_consensus.cc:2241] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
W20250629 01:52:00.658475 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9c1dc6d6d2b1458cb57875d2e25354bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c1dc6d6d2b1458cb57875d2e25354bc already in progress: creating tablet
I20250629 01:52:00.658540 1156 pending_rounds.cc:70] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:00.659322 1156 pending_rounds.cc:77] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759223031468032 op_type: NO_OP noop_request { }
I20250629 01:52:00.659418 1156 raft_consensus.cc:2887] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.659507 1156 raft_consensus.cc:2270] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.660202 1158 ts_tablet_manager.cc:1905] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.662626 449 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 8f9f59fbe9634ac8b27f990d9496f637 (table default.Table_1777ed0cebb04529b9bdc951af9addf4 [id=167bdf6c439f47ddb772b1b6c244473c]). Expected version 2 got 0
I20250629 01:52:00.665952 1156 ts_tablet_manager.cc:1905] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.672905 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ae366e6ccf34a9faf2caa2bded63546 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:00.673499 2580 raft_consensus.cc:491] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.673614 2580 raft_consensus.cc:513] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.674085 2580 leader_election.cc:290] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.675139 1155 tablet_bootstrap.cc:492] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.675328 1155 ts_tablet_manager.cc:1397] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.000s
I20250629 01:52:00.624393 2626 raft_consensus.cc:2747] T 2ad265c77d0047c38430242c53182e01 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250629 01:52:00.627400 1159 tablet_bootstrap.cc:492] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.676018 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "303f77a729c648f9b1146d11056b242d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.676208 1159 ts_tablet_manager.cc:1397] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.113s user 0.001s sys 0.009s
I20250629 01:52:00.676184 777 raft_consensus.cc:2466] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:00.634779 2625 raft_consensus.cc:491] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.676391 2625 raft_consensus.cc:513] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.676038 1155 raft_consensus.cc:357] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.676793 1155 raft_consensus.cc:383] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.676805 1159 raft_consensus.cc:357] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.676873 1155 raft_consensus.cc:738] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.676879 2625 leader_election.cc:290] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:00.676928 1159 raft_consensus.cc:383] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.676971 1159 raft_consensus.cc:738] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.677037 1155 consensus_queue.cc:260] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.677135 1159 consensus_queue.cc:260] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.677438 1155 ts_tablet_manager.cc:1428] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:52:00.677531 1159 ts_tablet_manager.cc:1428] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:00.677733 1159 tablet_bootstrap.cc:492] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.677994 986 leader_election.cc:304] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.678177 2580 raft_consensus.cc:2802] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:00.678237 2580 raft_consensus.cc:491] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.678282 2580 raft_consensus.cc:3058] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.679028 1159 tablet_bootstrap.cc:654] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.679858 1155 tablet_bootstrap.cc:492] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.680447 2580 raft_consensus.cc:513] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.681017 1155 tablet_bootstrap.cc:654] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.681299 2580 leader_election.cc:290] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.682143 2526 tablet_replica.cc:331] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:00.682286 2526 raft_consensus.cc:2241] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.682381 2526 raft_consensus.cc:2270] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.682986 2526 ts_tablet_manager.cc:1905] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:00.677742 850 leader_election.cc:343] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:52:00.687247 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0ba9eabd06174b87a1485f4921895c5f (table default.Table_6e4d952c729c4984a21f16da686e28f0 [id=2a01904ec4bd4b48b528d7c0cfca4a43]) successfully deleted
W20250629 01:52:00.687431 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c969377aaf034e9b896c1b53366555ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c969377aaf034e9b896c1b53366555ff already in progress: creating tablet
W20250629 01:52:00.687613 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet bbe1f8083a884cdd97afce83bd22073c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe1f8083a884cdd97afce83bd22073c already in progress: creating tablet
I20250629 01:52:00.687637 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "303f77a729c648f9b1146d11056b242d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:00.689265 988 leader_election.cc:343] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:00.691754 850 leader_election.cc:343] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:52:00.691843 850 leader_election.cc:304] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.692132 2625 raft_consensus.cc:2747] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.696758 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 6b4d4d6dacb84f80903522d20de4e795 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:00.699560 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "303f77a729c648f9b1146d11056b242d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:00.699697 777 raft_consensus.cc:3058] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.701967 777 raft_consensus.cc:2466] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:00.702764 986 leader_election.cc:304] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:00.703042 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0365d5ad654a7a9df622a27bfbe6fd consensus state has no leader
I20250629 01:52:00.703390 2580 raft_consensus.cc:2802] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.703486 2580 raft_consensus.cc:695] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:00.703723 2580 consensus_queue.cc:237] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.707427 1159 tablet_bootstrap.cc:492] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.707566 1159 ts_tablet_manager.cc:1397] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20250629 01:52:00.708182 1159 raft_consensus.cc:357] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.708369 1159 raft_consensus.cc:383] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.708444 1159 raft_consensus.cc:738] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.708632 1159 consensus_queue.cc:260] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.708813 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_884f76ae76564136b99617ba1c1ca0a8" } modify_external_catalogs: true
I20250629 01:52:00.708943 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_884f76ae76564136b99617ba1c1ca0a8" } modify_external_catalogs: true
I20250629 01:52:00.709019 1159 ts_tablet_manager.cc:1428] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:00.709527 1159 tablet_bootstrap.cc:492] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.711110 1159 tablet_bootstrap.cc:654] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.637603 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a9d2aa33a5c4d9ebecaa83c32b39dcb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.674567 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "303f77a729c648f9b1146d11056b242d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.676280 2626 raft_consensus.cc:491] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.715051 2626 raft_consensus.cc:513] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.715538 2626 leader_election.cc:290] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:00.716001 988 leader_election.cc:343] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.716629 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9f59fbe9634ac8b27f990d9496f637" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:00.717759 988 consensus_peers.cc:489] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:00.718118 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_98056b52f364497899c1998d23f58ce7" } modify_external_catalogs: true
I20250629 01:52:00.718242 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_98056b52f364497899c1998d23f58ce7" } modify_external_catalogs: true
I20250629 01:52:00.720708 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f9f59fbe9634ac8b27f990d9496f637" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.721218 850 leader_election.cc:304] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.721557 2626 raft_consensus.cc:2747] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.722055 777 raft_consensus.cc:1273] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:52:00.722518 2527 consensus_queue.cc:1035] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:00.723737 1159 tablet_bootstrap.cc:492] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.723850 1159 ts_tablet_manager.cc:1397] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20250629 01:52:00.724634 1159 raft_consensus.cc:357] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.724757 1159 raft_consensus.cc:383] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.724805 1159 raft_consensus.cc:738] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.724982 1159 consensus_queue.cc:260] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.725441 1159 ts_tablet_manager.cc:1428] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:00.725692 1159 tablet_bootstrap.cc:492] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.726433 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dcbcbed65c4445e7843620142b602552" } modify_external_catalogs: true
I20250629 01:52:00.726589 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_dcbcbed65c4445e7843620142b602552" } modify_external_catalogs: true
I20250629 01:52:00.727144 1159 tablet_bootstrap.cc:654] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.731499 2527 raft_consensus.cc:491] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.731632 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e3874404c3444d47a71f1dce7fb86ff8
I20250629 01:52:00.731637 2527 raft_consensus.cc:513] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.732251 2527 leader_election.cc:290] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.732553 892 tablet_service.cc:1515] Processing DeleteTablet for tablet e3874404c3444d47a71f1dce7fb86ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.733186 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet e3874404c3444d47a71f1dce7fb86ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.733165 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4d6f1eb43b24fda94aa0afcf4501195" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.733327 774 raft_consensus.cc:2466] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:00.733749 986 leader_election.cc:304] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.733794 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4d6f1eb43b24fda94aa0afcf4501195" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:00.734077 2675 raft_consensus.cc:2802] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250629 01:52:00.734124 988 leader_election.cc:343] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.734179 2675 raft_consensus.cc:491] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:00.734248 2675 raft_consensus.cc:3058] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.735116 752 tablet_service.cc:1515] Processing DeleteTablet for tablet e3874404c3444d47a71f1dce7fb86ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.735563 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c0c3a2b64e794d63a3c9bce071cf5b97" } new_table_name: "default.Table_892a1c3fe5b6433bab9788f706a4dd74" modify_external_catalogs: true
I20250629 01:52:00.736995 2675 raft_consensus.cc:513] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.738685 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4d6f1eb43b24fda94aa0afcf4501195" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:00.738806 774 raft_consensus.cc:3058] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:00.741070 774 raft_consensus.cc:2466] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:00.741524 986 leader_election.cc:304] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:00.741768 2527 raft_consensus.cc:2802] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:00.746819 2626 raft_consensus.cc:491] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.746945 2626 raft_consensus.cc:513] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.747548 2626 leader_election.cc:290] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:00.747985 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:00.748483 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bc3d470df0d4814a655bcff68c956e2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:00.748844 850 leader_election.cc:304] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.749110 2626 raft_consensus.cc:2747] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.750372 1159 tablet_bootstrap.cc:492] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:00.751013 1159 ts_tablet_manager.cc:1397] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
I20250629 01:52:00.751230 2675 leader_election.cc:290] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:00.752031 1159 raft_consensus.cc:357] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.752947 2527 raft_consensus.cc:695] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:00.752161 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e4d6f1eb43b24fda94aa0afcf4501195" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:00.753240 2527 consensus_queue.cc:237] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:00.753479 988 leader_election.cc:343] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.775585 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8ae366e6ccf34a9faf2caa2bded63546
I20250629 01:52:00.776019 454 catalog_manager.cc:6652] ReplaceTablet: tablet 8ae366e6ccf34a9faf2caa2bded63546 deleted and replaced by tablet 9370ec79715e4e5fa4766f202765f04b
I20250629 01:52:00.776208 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 7edfdf771ecd4ae8b87791846536271f from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:00.776365 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae366e6ccf34a9faf2caa2bded63546 with delete_type TABLET_DATA_DELETED (replaced by tablet 9370ec79715e4e5fa4766f202765f04b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.776891 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae366e6ccf34a9faf2caa2bded63546 with delete_type TABLET_DATA_DELETED (replaced by tablet 9370ec79715e4e5fa4766f202765f04b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.753157 1159 raft_consensus.cc:383] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.778698 1159 raft_consensus.cc:738] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.778913 1159 consensus_queue.cc:260] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.779354 1159 ts_tablet_manager.cc:1428] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20250629 01:52:00.780280 2578 tablet.cc:1722] T 3618e21295a84926bda02aa1d8e90a32 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.781816 1154 tablet_bootstrap.cc:492] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.781981 1154 ts_tablet_manager.cc:1397] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.145s user 0.026s sys 0.018s
I20250629 01:52:00.782500 1159 tablet_bootstrap.cc:492] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:00.782735 1154 raft_consensus.cc:357] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.782923 1154 raft_consensus.cc:383] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.783005 1154 raft_consensus.cc:738] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.783236 1154 consensus_queue.cc:260] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.783710 1154 ts_tablet_manager.cc:1428] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:00.783769 1159 tablet_bootstrap.cc:654] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.783999 1154 tablet_bootstrap.cc:492] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.785408 1154 tablet_bootstrap.cc:654] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.792631 1158 ts_tablet_manager.cc:1918] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:00.792721 1158 log.cc:1199] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8975c1deaaf44761be791f460b53ec74
I20250629 01:52:00.792986 1158 ts_tablet_manager.cc:1939] T 8975c1deaaf44761be791f460b53ec74 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.793568 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1248) took 1577 ms. Trace:
I20250629 01:52:00.793643 1158 rpcz_store.cc:276] 0629 01:51:59.215823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.215867 (+ 44us) service_pool.cc:225] Handling call
0629 01:52:00.655821 (+1439954us) ts_tablet_manager.cc:1145] Deleting tablet 8975c1deaaf44761be791f460b53ec74
0629 01:52:00.655829 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.785445 (+129616us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.792569 (+ 7124us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.793554 (+ 985us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379029}
I20250629 01:52:00.793892 1158 tablet_replica.cc:331] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.794026 1158 raft_consensus.cc:2241] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.794130 1158 raft_consensus.cc:2270] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.794057 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8975c1deaaf44761be791f460b53ec74 (table default.Table_d532629a716e44368b4324cf8b41f873 [id=21d8cd304488494fb56540f29f71f25d]) successfully deleted
I20250629 01:52:00.794514 1158 ts_tablet_manager.cc:1905] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.796245 2670 raft_consensus.cc:491] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:00.796387 2670 raft_consensus.cc:513] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.796762 1154 tablet_bootstrap.cc:492] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.796904 1154 ts_tablet_manager.cc:1397] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
W20250629 01:52:00.798139 850 leader_election.cc:343] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:52:00.798309 2670 leader_election.cc:290] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:00.802933 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bae91d28f27f4e14a23c65e41091b671
I20250629 01:52:00.757330 1156 ts_tablet_manager.cc:1918] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250629 01:52:00.771337 712 leader_election.cc:343] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:00.803934 712 leader_election.cc:304] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:00.803968 1156 log.cc:1199] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5b8a2d1a7f7146f4968590914374ba23
I20250629 01:52:00.804121 742 tablet_service.cc:1515] Processing DeleteTablet for tablet bae91d28f27f4e14a23c65e41091b671 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.804147 2651 raft_consensus.cc:2747] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.804571 1156 ts_tablet_manager.cc:1939] T 5b8a2d1a7f7146f4968590914374ba23 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.804685 892 tablet_service.cc:1515] Processing DeleteTablet for tablet bae91d28f27f4e14a23c65e41091b671 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.805179 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet bae91d28f27f4e14a23c65e41091b671 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.805346 1156 tablet_replica.cc:331] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:00.805498 1156 raft_consensus.cc:2241] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.805545 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5b8a2d1a7f7146f4968590914374ba23 (table default.Table_98a1aa99dbcb4f448201bde47757f522 [id=5ee5e560e23a45ff897dbba0a0909cda]) successfully deleted
I20250629 01:52:00.805591 2578 tablet.cc:1722] T cd9a41fae477470281dfba4d4e552b99 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.805758 1156 raft_consensus.cc:2270] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.806231 1156 ts_tablet_manager.cc:1905] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:00.807114 850 leader_election.cc:343] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:52:00.807209 850 leader_election.cc:304] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:00.771930 1155 tablet_bootstrap.cc:492] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.808079 2487 tablet.cc:1722] T 3618e21295a84926bda02aa1d8e90a32 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.808274 1155 ts_tablet_manager.cc:1397] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.128s user 0.003s sys 0.000s
I20250629 01:52:00.777578 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ae366e6ccf34a9faf2caa2bded63546 with delete_type TABLET_DATA_DELETED (replaced by tablet 9370ec79715e4e5fa4766f202765f04b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.779057 2558 tablet.cc:1722] T 3618e21295a84926bda02aa1d8e90a32 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.808794 1155 raft_consensus.cc:357] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.809932 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_44ab2296b2644a9282db2646084a4c95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.810393 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44ab2296b2644a9282db2646084a4c95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.814534 1155 raft_consensus.cc:383] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.814957 1154 raft_consensus.cc:357] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.815140 1155 raft_consensus.cc:738] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.815402 1155 consensus_queue.cc:260] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.815930 1155 ts_tablet_manager.cc:1428] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
W20250629 01:52:00.815343 988 consensus_peers.cc:489] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:00.816139 1154 raft_consensus.cc:383] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.816248 1154 raft_consensus.cc:738] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.816462 1154 consensus_queue.cc:260] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.816787 1155 tablet_bootstrap.cc:492] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.818418 1154 ts_tablet_manager.cc:1428] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250629 01:52:00.818686 1154 tablet_bootstrap.cc:492] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.818939 2558 tablet.cc:1722] T cd9a41fae477470281dfba4d4e552b99 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:00.821300 1155 tablet_bootstrap.cc:654] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.823156 2670 raft_consensus.cc:2747] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:00.823977 1154 tablet_bootstrap.cc:654] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.824571 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c2591c65d53449c8801778606f1f05f1
I20250629 01:52:00.825466 892 tablet_service.cc:1515] Processing DeleteTablet for tablet c2591c65d53449c8801778606f1f05f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:00.826059 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c2591c65d53449c8801778606f1f05f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.826197 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c2591c65d53449c8801778606f1f05f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:00.830832 2526 ts_tablet_manager.cc:1918] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.830953 2526 log.cc:1199] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/1c995ce14c9a4966b842fcba520cf87d
I20250629 01:52:00.831243 2526 ts_tablet_manager.cc:1939] T 1c995ce14c9a4966b842fcba520cf87d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.831691 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 1c995ce14c9a4966b842fcba520cf87d (table default.Table_2ba661f263ce4cbda78e8be06f7d662d [id=89a6d7dc147e4ce6bb8efc45d1f69eb5]) successfully deleted
W20250629 01:52:00.832199 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 77f6bab8deb740cc9cf43832196e404c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f6bab8deb740cc9cf43832196e404c already in progress: creating tablet
W20250629 01:52:00.832729 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9c1dc6d6d2b1458cb57875d2e25354bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c1dc6d6d2b1458cb57875d2e25354bc already in progress: creating tablet
I20250629 01:52:00.832932 2526 tablet_replica.cc:331] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:00.833129 2526 raft_consensus.cc:2241] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:00.833271 2526 raft_consensus.cc:2270] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.833884 2526 ts_tablet_manager.cc:1905] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.839890 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3e8277c1c24c4c58aba9b1f245d490c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.840361 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e8277c1c24c4c58aba9b1f245d490c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.841356 1155 tablet_bootstrap.cc:492] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.841501 1155 ts_tablet_manager.cc:1397] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20250629 01:52:00.842087 1155 raft_consensus.cc:357] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.842283 1155 raft_consensus.cc:383] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.842370 1155 raft_consensus.cc:738] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.842589 1155 consensus_queue.cc:260] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.843134 1155 ts_tablet_manager.cc:1428] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:00.843613 1155 tablet_bootstrap.cc:492] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:00.845377 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bbff29d38bf437da3c8273ba1bbec3f consensus state has no leader
I20250629 01:52:00.847797 1155 tablet_bootstrap.cc:654] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.848197 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3730b225150645e09a3a9b396cb134a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.848609 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3730b225150645e09a3a9b396cb134a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.849206 1154 tablet_bootstrap.cc:492] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.849391 1154 ts_tablet_manager.cc:1397] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250629 01:52:00.850128 1154 raft_consensus.cc:357] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.850323 1154 raft_consensus.cc:383] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.850414 1154 raft_consensus.cc:738] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.850494 1155 tablet_bootstrap.cc:492] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.850600 1155 ts_tablet_manager.cc:1397] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:52:00.850716 1154 consensus_queue.cc:260] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.851143 1155 raft_consensus.cc:357] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.851274 1155 raft_consensus.cc:383] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.851323 1155 raft_consensus.cc:738] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.851475 1155 consensus_queue.cc:260] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.851857 1155 ts_tablet_manager.cc:1428] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:52:00.852067 1155 tablet_bootstrap.cc:492] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.853264 1155 tablet_bootstrap.cc:654] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.853603 1154 ts_tablet_manager.cc:1428] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:52:00.853880 1154 tablet_bootstrap.cc:492] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.855293 1154 tablet_bootstrap.cc:654] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.862416 2531 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: acquiring CA information for follower catalog manager: success
I20250629 01:52:00.853924 1158 ts_tablet_manager.cc:1918] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.862731 1158 log.cc:1199] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5b8a2d1a7f7146f4968590914374ba23
I20250629 01:52:00.863078 1158 ts_tablet_manager.cc:1939] T 5b8a2d1a7f7146f4968590914374ba23 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.863557 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dc29904c04dd44f9b8f1dc8bdf527229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:00.863754 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1249) took 1647 ms. Trace:
I20250629 01:52:00.863811 1158 rpcz_store.cc:276] 0629 01:51:59.216504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.216546 (+ 42us) service_pool.cc:225] Handling call
0629 01:52:00.793857 (+1577311us) ts_tablet_manager.cc:1145] Deleting tablet 5b8a2d1a7f7146f4968590914374ba23
0629 01:52:00.793864 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.847274 (+ 53410us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.853855 (+ 6581us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.863738 (+ 9883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517017}
W20250629 01:52:00.863952 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc29904c04dd44f9b8f1dc8bdf527229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.864152 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5b8a2d1a7f7146f4968590914374ba23 (table default.Table_98a1aa99dbcb4f448201bde47757f522 [id=5ee5e560e23a45ff897dbba0a0909cda]) successfully deleted
I20250629 01:52:00.864468 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1262) took 1269 ms. Trace:
I20250629 01:52:00.864547 1158 rpcz_store.cc:276] 0629 01:51:59.594834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.595912 (+ 1078us) service_pool.cc:225] Handling call
0629 01:52:00.864362 (+1268450us) ts_tablet_manager.cc:1145] Deleting tablet 7aef2514a87c4c4aaf4d51b314630185
0629 01:52:00.864369 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.864444 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1268260}
W20250629 01:52:00.864861 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7aef2514a87c4c4aaf4d51b314630185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7aef2514a87c4c4aaf4d51b314630185 already in progress: creating tablet
I20250629 01:52:00.865135 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1263) took 1269 ms. Trace:
I20250629 01:52:00.865207 1158 rpcz_store.cc:276] 0629 01:51:59.595873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.598813 (+ 2940us) service_pool.cc:225] Handling call
0629 01:52:00.865046 (+1266233us) ts_tablet_manager.cc:1145] Deleting tablet ab59d0a03543499fbacaf39743ca812b
0629 01:52:00.865054 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.865122 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266101}
W20250629 01:52:00.865511 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ab59d0a03543499fbacaf39743ca812b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab59d0a03543499fbacaf39743ca812b already in progress: creating tablet
I20250629 01:52:00.865784 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1265) took 1246 ms. Trace:
I20250629 01:52:00.865849 1158 rpcz_store.cc:276] 0629 01:51:59.619116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.620914 (+ 1798us) service_pool.cc:225] Handling call
0629 01:52:00.865700 (+1244786us) ts_tablet_manager.cc:1145] Deleting tablet 12d63924c5ff4a99ac62f89a9d29423b
0629 01:52:00.865707 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.865771 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1244653}
W20250629 01:52:00.866175 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 12d63924c5ff4a99ac62f89a9d29423b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12d63924c5ff4a99ac62f89a9d29423b already in progress: creating tablet
I20250629 01:52:00.869694 1155 tablet_bootstrap.cc:492] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.869799 1155 ts_tablet_manager.cc:1397] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250629 01:52:00.870327 1155 raft_consensus.cc:357] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.870458 1155 raft_consensus.cc:383] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.870502 1155 raft_consensus.cc:738] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.870628 1155 consensus_queue.cc:260] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.871009 1155 ts_tablet_manager.cc:1428] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:52:00.871222 1155 tablet_bootstrap.cc:492] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:00.872501 1155 tablet_bootstrap.cc:654] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.874150 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1264) took 1255 ms. Trace:
I20250629 01:52:00.874219 1158 rpcz_store.cc:276] 0629 01:51:59.618895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.621333 (+ 2438us) service_pool.cc:225] Handling call
0629 01:52:00.873428 (+1252095us) ts_tablet_manager.cc:1145] Deleting tablet f7e6f33c7bee44d1a2b6feb0625faedf
0629 01:52:00.873436 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.874135 (+ 699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251956}
I20250629 01:52:00.875536 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1266) took 1192 ms. Trace:
W20250629 01:52:00.875622 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f7e6f33c7bee44d1a2b6feb0625faedf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f7e6f33c7bee44d1a2b6feb0625faedf already in progress: creating tablet
I20250629 01:52:00.875597 1158 rpcz_store.cc:276] 0629 01:51:59.682618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.682656 (+ 38us) service_pool.cc:225] Handling call
0629 01:52:00.875441 (+1192785us) ts_tablet_manager.cc:1145] Deleting tablet 46bb76c63ce749078365420f224d62f3
0629 01:52:00.875449 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.875523 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1192307}
I20250629 01:52:00.875844 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1267) took 1186 ms. Trace:
I20250629 01:52:00.875903 1158 rpcz_store.cc:276] 0629 01:51:59.688950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.688993 (+ 43us) service_pool.cc:225] Handling call
0629 01:52:00.875760 (+1186767us) ts_tablet_manager.cc:1145] Deleting tablet 24e9caef62f64a4281789aba8f83386a
0629 01:52:00.875767 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.875827 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1186594}
I20250629 01:52:00.876142 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1268) took 1186 ms. Trace:
I20250629 01:52:00.876214 1158 rpcz_store.cc:276] 0629 01:51:59.689696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.689734 (+ 38us) service_pool.cc:225] Handling call
0629 01:52:00.876044 (+1186310us) ts_tablet_manager.cc:1145] Deleting tablet c4c4153b6dcd45ada716c05d0590d10a
0629 01:52:00.876051 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.876127 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1186123}
I20250629 01:52:00.876454 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1270) took 1163 ms. Trace:
I20250629 01:52:00.876529 1158 rpcz_store.cc:276] 0629 01:51:59.713398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.713443 (+ 45us) service_pool.cc:225] Handling call
0629 01:52:00.876370 (+1162927us) ts_tablet_manager.cc:1145] Deleting tablet 504c15b3b168499d9a82584f82a0b792
0629 01:52:00.876378 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.876438 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162764}
I20250629 01:52:00.876535 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:00.876770 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1271) took 1142 ms. Trace:
I20250629 01:52:00.876822 1158 rpcz_store.cc:276] 0629 01:51:59.734588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.734630 (+ 42us) service_pool.cc:225] Handling call
0629 01:52:00.876694 (+1142064us) ts_tablet_manager.cc:1145] Deleting tablet 8ae0d61d254c4e1ba508e30aa35600ca
0629 01:52:00.876700 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.876757 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1141905}
W20250629 01:52:00.876889 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.877200 1158 tablet_replica.cc:331] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:00.877355 1158 raft_consensus.cc:2241] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
W20250629 01:52:00.877880 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 46bb76c63ce749078365420f224d62f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46bb76c63ce749078365420f224d62f3 already in progress: creating tablet
W20250629 01:52:00.878033 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 24e9caef62f64a4281789aba8f83386a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24e9caef62f64a4281789aba8f83386a already in progress: creating tablet
W20250629 01:52:00.878150 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c4c4153b6dcd45ada716c05d0590d10a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4c4153b6dcd45ada716c05d0590d10a already in progress: creating tablet
W20250629 01:52:00.878266 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 504c15b3b168499d9a82584f82a0b792 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 504c15b3b168499d9a82584f82a0b792 already in progress: creating tablet
W20250629 01:52:00.878376 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8ae0d61d254c4e1ba508e30aa35600ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae0d61d254c4e1ba508e30aa35600ca already in progress: creating tablet
W20250629 01:52:00.878504 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 77f6bab8deb740cc9cf43832196e404c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77f6bab8deb740cc9cf43832196e404c already in progress: creating tablet
I20250629 01:52:00.884300 1158 pending_rounds.cc:70] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:00.884354 1158 pending_rounds.cc:77] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759224109088768 op_type: NO_OP noop_request { }
I20250629 01:52:00.884440 1158 raft_consensus.cc:2887] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:00.884529 1158 raft_consensus.cc:2270] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:00.886240 1158 ts_tablet_manager.cc:1905] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:00.888573 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c24aa328eefc49d6979d74d3945cc219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.888983 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c24aa328eefc49d6979d74d3945cc219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.894871 1154 tablet_bootstrap.cc:492] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.894974 1154 ts_tablet_manager.cc:1397] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.011s
I20250629 01:52:00.895597 1154 raft_consensus.cc:357] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.895735 1154 raft_consensus.cc:383] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.895790 1154 raft_consensus.cc:738] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.895952 1154 consensus_queue.cc:260] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.896380 1154 ts_tablet_manager.cc:1428] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:52:00.896346 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a5007857174542ea9967e05f32de9d93" } new_table_name: "default.Table_2831a79356254e00b88ca2149d61af2c" modify_external_catalogs: true
I20250629 01:52:00.896790 1154 tablet_bootstrap.cc:492] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.898164 1154 tablet_bootstrap.cc:654] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.898561 1158 ts_tablet_manager.cc:1918] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:00.898638 1158 log.cc:1199] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/45ca747f03524076b4112e4db78614fe
I20250629 01:52:00.898890 1158 ts_tablet_manager.cc:1939] T 45ca747f03524076b4112e4db78614fe P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:00.900789 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 45ca747f03524076b4112e4db78614fe (table default.Table_e39f46841d454041bb064e74d3c27bc0 [id=f7eba54966074342a3632852c544ff35]) successfully deleted
W20250629 01:52:00.901182 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c969377aaf034e9b896c1b53366555ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c969377aaf034e9b896c1b53366555ff already in progress: creating tablet
W20250629 01:52:00.901700 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet bbe1f8083a884cdd97afce83bd22073c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bbe1f8083a884cdd97afce83bd22073c already in progress: creating tablet
W20250629 01:52:00.901880 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9c1dc6d6d2b1458cb57875d2e25354bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c1dc6d6d2b1458cb57875d2e25354bc already in progress: creating tablet
W20250629 01:52:00.902040 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e4ff12d702bf48039d63c437079979fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ff12d702bf48039d63c437079979fd already in progress: creating tablet
W20250629 01:52:00.902204 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 24c4841e127b495cbfbca0103662c01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4841e127b495cbfbca0103662c01e already in progress: creating tablet
W20250629 01:52:00.902372 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 27b76e04158e492a9979c06e7cc2ae37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b76e04158e492a9979c06e7cc2ae37 already in progress: creating tablet
W20250629 01:52:00.902531 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4b11d32f1c3e41f3b9f6d20fd837226a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b11d32f1c3e41f3b9f6d20fd837226a already in progress: creating tablet
W20250629 01:52:00.902972 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9c66df5169224fea8c4fc45df7d695d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c66df5169224fea8c4fc45df7d695d8 already in progress: creating tablet
W20250629 01:52:00.903134 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet f93f07e50b414db29c081760ce2004c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f93f07e50b414db29c081760ce2004c1 already in progress: creating tablet
W20250629 01:52:00.903266 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet b5bcfb2ac1674beb8e9d50d2818cec0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bcfb2ac1674beb8e9d50d2818cec0d already in progress: creating tablet
W20250629 01:52:00.903376 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e3874404c3444d47a71f1dce7fb86ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3874404c3444d47a71f1dce7fb86ff8 already in progress: creating tablet
W20250629 01:52:00.903487 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 8ae366e6ccf34a9faf2caa2bded63546 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae366e6ccf34a9faf2caa2bded63546 already in progress: creating tablet
W20250629 01:52:00.903625 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet bae91d28f27f4e14a23c65e41091b671 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bae91d28f27f4e14a23c65e41091b671 already in progress: creating tablet
W20250629 01:52:00.903779 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c2591c65d53449c8801778606f1f05f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2591c65d53449c8801778606f1f05f1 already in progress: creating tablet
I20250629 01:52:00.926810 1154 tablet_bootstrap.cc:492] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.926980 1154 ts_tablet_manager.cc:1397] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.005s
I20250629 01:52:00.927572 1154 raft_consensus.cc:357] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.927747 1154 raft_consensus.cc:383] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.927819 1154 raft_consensus.cc:738] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.928015 1154 consensus_queue.cc:260] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.930604 1154 ts_tablet_manager.cc:1428] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:52:00.930871 1154 tablet_bootstrap.cc:492] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.932852 1154 tablet_bootstrap.cc:654] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.936570 2531 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 0f4258c004e04bcf8216bb7c8fd56c54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250629 01:52:00.939126 1154 tablet_bootstrap.cc:492] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:00.939277 1154 ts_tablet_manager.cc:1397] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20250629 01:52:00.939875 1154 raft_consensus.cc:357] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.940165 1154 raft_consensus.cc:383] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:00.940255 1154 raft_consensus.cc:738] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:00.940455 1154 consensus_queue.cc:260] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:00.940893 1154 ts_tablet_manager.cc:1428] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:00.941179 1154 tablet_bootstrap.cc:492] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:00.943012 1154 tablet_bootstrap.cc:654] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:00.960325 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_54de7744e36d4183a8af975cdc699bd3" } new_table_name: "default.Table_5d53a01d5be7418c988913852a7f91a3" modify_external_catalogs: true
I20250629 01:52:00.964010 2548 consensus_queue.cc:1035] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:00.965947 453 master_service.cc:948] ReplaceTablet: received request to replace tablet adbb648c49e24b8085eacbe6e0a86991 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:00.981567 1156 ts_tablet_manager.cc:1918] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.981673 1156 log.cc:1199] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8975c1deaaf44761be791f460b53ec74
I20250629 01:52:00.981922 1156 ts_tablet_manager.cc:1939] T 8975c1deaaf44761be791f460b53ec74 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:00.983052 1155 tablet_bootstrap.cc:492] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:00.983201 1155 ts_tablet_manager.cc:1397] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.018s
I20250629 01:52:00.983862 1155 raft_consensus.cc:357] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:00.987960 2526 ts_tablet_manager.cc:1918] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:00.988060 2526 log.cc:1199] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/56b60431b72e4792aa1c32fdee40cdb9
I20250629 01:52:00.988408 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8975c1deaaf44761be791f460b53ec74 (table default.Table_d532629a716e44368b4324cf8b41f873 [id=21d8cd304488494fb56540f29f71f25d]) successfully deleted
W20250629 01:52:00.988595 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e4ff12d702bf48039d63c437079979fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ff12d702bf48039d63c437079979fd already in progress: creating tablet
W20250629 01:52:00.989184 712 consensus_peers.cc:489] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:00.990440 1155 raft_consensus.cc:383] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:52:00.992555 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 999dae3757ad4c57bf4e2a92ee5203ca consensus state has no leader
I20250629 01:52:00.992830 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_952ee4fb7e584b3cb71d943396d62c65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.993213 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_952ee4fb7e584b3cb71d943396d62c65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.997524 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet adbb648c49e24b8085eacbe6e0a86991
I20250629 01:52:00.997848 453 catalog_manager.cc:6652] ReplaceTablet: tablet adbb648c49e24b8085eacbe6e0a86991 deleted and replaced by tablet 059b0999c6114f4681be7b97cd9eb071
I20250629 01:52:00.998077 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b614c356b8dd4e5d854d84b62f1ff989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:00.998421 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b614c356b8dd4e5d854d84b62f1ff989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:00.999662 2526 ts_tablet_manager.cc:1939] T 56b60431b72e4792aa1c32fdee40cdb9 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:00.999941 2526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1249) took 1024 ms. Trace:
I20250629 01:52:01.000025 2526 rpcz_store.cc:276] 0629 01:51:59.975346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.976141 (+ 795us) service_pool.cc:225] Handling call
0629 01:52:00.832887 (+856746us) ts_tablet_manager.cc:1145] Deleting tablet 56b60431b72e4792aa1c32fdee40cdb9
0629 01:52:00.832897 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:00.925813 (+ 92916us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.987878 (+ 62065us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:00.999924 (+ 12046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":856374}
W20250629 01:52:01.000341 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 24c4841e127b495cbfbca0103662c01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4841e127b495cbfbca0103662c01e already in progress: creating tablet
I20250629 01:52:01.000377 2526 tablet_replica.cc:331] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:52:01.000519 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 27b76e04158e492a9979c06e7cc2ae37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b76e04158e492a9979c06e7cc2ae37 already in progress: creating tablet
I20250629 01:52:01.000581 2526 raft_consensus.cc:2241] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:52:01.000681 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4b11d32f1c3e41f3b9f6d20fd837226a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b11d32f1c3e41f3b9f6d20fd837226a already in progress: creating tablet
I20250629 01:52:01.000706 2526 raft_consensus.cc:2270] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:01.000819 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 9c66df5169224fea8c4fc45df7d695d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c66df5169224fea8c4fc45df7d695d8 already in progress: creating tablet
W20250629 01:52:01.000953 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet f93f07e50b414db29c081760ce2004c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f93f07e50b414db29c081760ce2004c1 already in progress: creating tablet
I20250629 01:52:01.000914 892 tablet_service.cc:1515] Processing DeleteTablet for tablet adbb648c49e24b8085eacbe6e0a86991 with delete_type TABLET_DATA_DELETED (replaced by tablet 059b0999c6114f4681be7b97cd9eb071) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.001202 2526 ts_tablet_manager.cc:1905] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.002956 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet adbb648c49e24b8085eacbe6e0a86991 with delete_type TABLET_DATA_DELETED (replaced by tablet 059b0999c6114f4681be7b97cd9eb071) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.003135 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 56b60431b72e4792aa1c32fdee40cdb9 (table default.Table_67bf8a3296bd47d7bf3977d6d088689c [id=7749e44179bf4424bbcb595803f61dae]) successfully deleted
I20250629 01:52:01.003193 1158 tablet_replica.cc:331] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.003363 1158 raft_consensus.cc:2241] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.003495 1158 raft_consensus.cc:2270] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.003944 1158 ts_tablet_manager.cc:1905] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.005872 1159 tablet_bootstrap.cc:492] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.006026 1159 ts_tablet_manager.cc:1397] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.224s user 0.008s sys 0.033s
I20250629 01:52:01.006805 1159 raft_consensus.cc:357] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.006999 1159 raft_consensus.cc:383] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.007086 1159 raft_consensus.cc:738] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.007308 1159 consensus_queue.cc:260] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.007747 1159 ts_tablet_manager.cc:1428] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:01.009738 1158 ts_tablet_manager.cc:1918] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.009872 1158 log.cc:1199] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/adbb648c49e24b8085eacbe6e0a86991
I20250629 01:52:01.010198 1158 ts_tablet_manager.cc:1939] T adbb648c49e24b8085eacbe6e0a86991 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.011354 1159 tablet_bootstrap.cc:492] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.013500 1154 tablet_bootstrap.cc:492] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.013609 1154 ts_tablet_manager.cc:1397] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.045s
I20250629 01:52:01.014181 1154 raft_consensus.cc:357] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.014318 1154 raft_consensus.cc:383] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.014367 1154 raft_consensus.cc:738] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.014529 1154 consensus_queue.cc:260] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.014797 1159 tablet_bootstrap.cc:654] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.016064 1155 raft_consensus.cc:738] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.016418 1155 consensus_queue.cc:260] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.018504 1155 ts_tablet_manager.cc:1428] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.035s user 0.001s sys 0.000s
I20250629 01:52:01.018716 1154 ts_tablet_manager.cc:1428] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250629 01:52:01.002455 752 tablet_service.cc:1515] Processing DeleteTablet for tablet adbb648c49e24b8085eacbe6e0a86991 with delete_type TABLET_DATA_DELETED (replaced by tablet 059b0999c6114f4681be7b97cd9eb071) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:00.999565 1156 tablet_replica.cc:331] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:52:01.019551 988 consensus_peers.cc:489] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:01.021767 1154 tablet_bootstrap.cc:492] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.023064 1154 tablet_bootstrap.cc:654] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.011137 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet adbb648c49e24b8085eacbe6e0a86991 (table default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]) successfully deleted
I20250629 01:52:01.025354 2049 raft_consensus.cc:491] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.025468 2049 raft_consensus.cc:513] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.025993 2049 leader_election.cc:290] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.026619 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50d23f589de84dd5b150a17cee00c411" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.026778 1048 raft_consensus.cc:2466] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:01.026854 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50d23f589de84dd5b150a17cee00c411" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.027149 710 leader_election.cc:304] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:52:01.027292 712 leader_election.cc:343] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.027456 2049 raft_consensus.cc:2802] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.027519 2049 raft_consensus.cc:491] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.027571 2049 raft_consensus.cc:3058] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.028861 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3e0a0e61a2314bdba910cdd875e456bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.029232 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e0a0e61a2314bdba910cdd875e456bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.030437 2049 raft_consensus.cc:513] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.031184 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50d23f589de84dd5b150a17cee00c411" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.031256 2049 leader_election.cc:290] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.031327 1048 raft_consensus.cc:3058] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.031641 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50d23f589de84dd5b150a17cee00c411" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:01.031982 712 leader_election.cc:343] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.033669 1156 raft_consensus.cc:2241] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.033882 1156 pending_rounds.cc:70] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:01.033864 1048 raft_consensus.cc:2466] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:01.033929 1156 pending_rounds.cc:77] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759226226368512 op_type: NO_OP noop_request { }
I20250629 01:52:01.034003 1156 raft_consensus.cc:2887] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.034073 1156 raft_consensus.cc:2270] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.034384 710 leader_election.cc:304] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.034761 2049 raft_consensus.cc:2802] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.034976 2049 raft_consensus.cc:695] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.035219 2049 consensus_queue.cc:237] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.034761 1156 ts_tablet_manager.cc:1905] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.036582 1154 tablet_bootstrap.cc:492] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.036726 1154 ts_tablet_manager.cc:1397] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.001s
I20250629 01:52:01.037379 1154 raft_consensus.cc:357] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.037564 1154 raft_consensus.cc:383] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.037734 1154 raft_consensus.cc:738] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.037935 1154 consensus_queue.cc:260] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.038218 1155 tablet_bootstrap.cc:492] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.040076 1154 ts_tablet_manager.cc:1428] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:52:01.040341 1154 tablet_bootstrap.cc:492] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.041779 1154 tablet_bootstrap.cc:654] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.056010 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_27f4e21d29ed426089885ef45a154384" } new_table_name: "default.Table_a90148b3c1704f9db343e6bec329546e" modify_external_catalogs: true
I20250629 01:52:01.057775 2526 ts_tablet_manager.cc:1918] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.057876 2526 log.cc:1199] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/0d4320f615464fedbcab25137952f684
I20250629 01:52:01.058132 2526 ts_tablet_manager.cc:1939] T 0d4320f615464fedbcab25137952f684 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.062026 1156 ts_tablet_manager.cc:1918] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.064251 2526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1250) took 1088 ms. Trace:
W20250629 01:52:01.064275 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a873bc03533a405185eca6ad4bb5db3a consensus state has no leader
I20250629 01:52:01.064333 2526 rpcz_store.cc:276] 0629 01:51:59.975445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:51:59.976288 (+ 843us) service_pool.cc:225] Handling call
0629 01:52:01.000336 (+1024048us) ts_tablet_manager.cc:1145] Deleting tablet 0d4320f615464fedbcab25137952f684
0629 01:52:01.000345 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.025616 (+ 25271us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.057694 (+ 32078us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.064237 (+ 6543us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1023959}
I20250629 01:52:01.064803 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 0d4320f615464fedbcab25137952f684 (table default.Table_c7d1c2fe02d04423803f16f69f0d6625 [id=5a75eb0020fb4d4fb1d2066ed7d670a5]) successfully deleted
I20250629 01:52:01.064857 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4adf3f46833048789a2485d973c3bf59" } new_table_name: "default.Table_389aaf907bdc47ff873cdf71e7f26fad" modify_external_catalogs: true
I20250629 01:52:01.065141 2526 tablet_replica.cc:331] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.065336 2526 raft_consensus.cc:2241] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.070782 2670 raft_consensus.cc:491] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.070894 2670 raft_consensus.cc:513] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.071406 2670 leader_election.cc:290] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:01.073426 850 leader_election.cc:343] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 9cfff25bea764e62954e49611b71703c
W20250629 01:52:01.073787 850 leader_election.cc:343] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9cfff25bea764e62954e49611b71703c
I20250629 01:52:01.073872 850 leader_election.cc:304] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.074198 2670 raft_consensus.cc:2747] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.075457 446 catalog_manager.cc:5582] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:01.075769 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:01.076188 446 catalog_manager.cc:5582] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:01.076503 446 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:52:01.076711 892 tablet_service.cc:1468] Processing CreateTablet for tablet 6464fd2fa5bf46dabf3a177d5e756b94 (DEFAULT_TABLE table=default.Table_b9cb2b777214494e9a98a81d9b7785ee [id=9b580131b1d943d6b74bf75268435702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.076712 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 6464fd2fa5bf46dabf3a177d5e756b94 (DEFAULT_TABLE table=default.Table_b9cb2b777214494e9a98a81d9b7785ee [id=9b580131b1d943d6b74bf75268435702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.077023 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6464fd2fa5bf46dabf3a177d5e756b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.077021 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6464fd2fa5bf46dabf3a177d5e756b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.077344 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_3c4a230556474a8e831d7180634f0155 [id=297fd4ccf3cb46c283189b67c1b8e648] alter complete (version 2)
I20250629 01:52:01.077787 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a014907d09c94745859436f1eb73b280
I20250629 01:52:01.078329 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f3cd4aaddc584fb39053963cf9005c3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.078753 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3cd4aaddc584fb39053963cf9005c3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.078876 877 tablet_service.cc:1515] Processing DeleteTablet for tablet a014907d09c94745859436f1eb73b280 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.079583 449 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_61a4141b84a64b7085c02660cccf1cfd [id=e991e29b3d9d4dcc96912754565336f8] alter complete (version 2)
I20250629 01:52:01.079775 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 4dd23f6818f64b989dbb7aaaabcd816d (DEFAULT_TABLE table=default.Table_346dcc9d7d4c43fb8bc4573429d3aaa0 [id=be90c29ecc80438ead822197bf4f202c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.080021 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd23f6818f64b989dbb7aaaabcd816d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.080001 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aaa30d8b97a8406fb1f5fe4c5f309010" } modify_external_catalogs: true
I20250629 01:52:01.080128 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aaa30d8b97a8406fb1f5fe4c5f309010" } modify_external_catalogs: true
I20250629 01:52:01.080302 896 tablet_service.cc:1468] Processing CreateTablet for tablet 4dd23f6818f64b989dbb7aaaabcd816d (DEFAULT_TABLE table=default.Table_346dcc9d7d4c43fb8bc4573429d3aaa0 [id=be90c29ecc80438ead822197bf4f202c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.080560 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd23f6818f64b989dbb7aaaabcd816d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.083676 452 master_service.cc:948] ReplaceTablet: received request to replace tablet c5e9646c3ad04ed580f1d3402010dd38 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:01.084067 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b1e075cbb0dc4e5eb8d4de302ab08610" } modify_external_catalogs: true
I20250629 01:52:01.084200 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b1e075cbb0dc4e5eb8d4de302ab08610" } modify_external_catalogs: true
I20250629 01:52:01.084290 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a014907d09c94745859436f1eb73b280 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.084738 1158 tablet_replica.cc:331] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.084923 1158 raft_consensus.cc:2241] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.085071 1158 raft_consensus.cc:2270] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.085492 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b0782c7e652419096710563930f2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.085521 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.085628 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.085677 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.085785 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.085863 2527 raft_consensus.cc:491] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.085927 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.085973 2527 raft_consensus.cc:513] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.086195 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.086478 2527 leader_election.cc:290] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.087271 1158 ts_tablet_manager.cc:1905] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.088243 2526 raft_consensus.cc:2270] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.088663 2526 ts_tablet_manager.cc:1905] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.090162 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8295e8178eff4d5eb95ac885092731e0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.090274 988 consensus_peers.cc:489] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:01.090672 988 leader_election.cc:343] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.092832 892 tablet_service.cc:1468] Processing CreateTablet for tablet b17c4768b35443cf8c289a96f2cf6d13 (DEFAULT_TABLE table=default.Table_26700fd18fbf4cddb81d599cf016884c [id=6e590d1c3cf7433280ae8c12f57fde40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.093139 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17c4768b35443cf8c289a96f2cf6d13. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:01.093171 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 790a1347e57f4cea895132564eafae9c consensus state has no leader
I20250629 01:52:01.093279 1158 ts_tablet_manager.cc:1918] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.094765 1158 log.cc:1199] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a014907d09c94745859436f1eb73b280
I20250629 01:52:01.095031 1158 ts_tablet_manager.cc:1939] T a014907d09c94745859436f1eb73b280 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.095664 1158 tablet_replica.cc:331] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.095818 1158 raft_consensus.cc:2241] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.095999 1158 pending_rounds.cc:70] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:01.096042 1158 pending_rounds.cc:77] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759224946221056 op_type: NO_OP noop_request { }
I20250629 01:52:01.096364 1158 raft_consensus.cc:2887] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.096441 1158 raft_consensus.cc:2270] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.097012 1158 ts_tablet_manager.cc:1905] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.097368 2487 tablet.cc:1722] T d456410cd09845dabd979c123f70741b P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:52:01.101018 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a014907d09c94745859436f1eb73b280 (table default.Table_98056b52f364497899c1998d23f58ce7 [id=d63189e5f0564d20a7f992ec881309fd]) successfully deleted
I20250629 01:52:01.101655 1158 ts_tablet_manager.cc:1918] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.101751 1158 log.cc:1199] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/1b0782c7e652419096710563930f2072
I20250629 01:52:01.102017 1158 ts_tablet_manager.cc:1939] T 1b0782c7e652419096710563930f2072 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.102706 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 1b0782c7e652419096710563930f2072 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
I20250629 01:52:01.104502 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8a9b6060d8cd47aa8e1019d961b2fbab" } modify_external_catalogs: true
I20250629 01:52:01.104624 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8a9b6060d8cd47aa8e1019d961b2fbab" } modify_external_catalogs: true
W20250629 01:52:01.104648 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9fe9674ffeb44ba6ab60de59d4c08670 because the tablet was not found. No further retry: Not found: Tablet not found: 9fe9674ffeb44ba6ab60de59d4c08670
I20250629 01:52:01.105026 1158 tablet_replica.cc:331] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.105178 1158 raft_consensus.cc:2241] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.105316 1158 raft_consensus.cc:2270] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.106040 1023 tablet_service.cc:1468] Processing CreateTablet for tablet b17c4768b35443cf8c289a96f2cf6d13 (DEFAULT_TABLE table=default.Table_26700fd18fbf4cddb81d599cf016884c [id=6e590d1c3cf7433280ae8c12f57fde40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.106326 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17c4768b35443cf8c289a96f2cf6d13. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:01.107518 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 999dae3757ad4c57bf4e2a92ee5203ca (table default.Table_5d53a01d5be7418c988913852a7f91a3 [id=194bdf24776b480895832c6d00becb73], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 999dae3757ad4c57bf4e2a92ee5203ca consensus state has no leader
I20250629 01:52:01.107595 2526 ts_tablet_manager.cc:1918] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.107667 2526 log.cc:1199] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/2845d8fbf3f34c769027b9b6bb3e1f08
I20250629 01:52:01.107885 2526 ts_tablet_manager.cc:1939] T 2845d8fbf3f34c769027b9b6bb3e1f08 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.110734 2578 tablet.cc:1722] T d456410cd09845dabd979c123f70741b P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:52:01.113330 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2213f74c54534046a671808db21f6ee8 (DEFAULT_TABLE table=default.Table_073d8d6803d541acb4e3549a8ca2dca2 [id=161db0d2090f4f6faf8122fddd3942d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.113692 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2213f74c54534046a671808db21f6ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.114202 877 tablet_service.cc:1468] Processing CreateTablet for tablet 0fc89468158e44a3bb604a590b07acf6 (DEFAULT_TABLE table=default.Table_fb78bf13072345fea20a096dffdba5fb [id=d88ad526d4344361b6081a2e8fc126e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.114495 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fc89468158e44a3bb604a590b07acf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.114686 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2213f74c54534046a671808db21f6ee8 (DEFAULT_TABLE table=default.Table_073d8d6803d541acb4e3549a8ca2dca2 [id=161db0d2090f4f6faf8122fddd3942d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.114957 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2213f74c54534046a671808db21f6ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.077286 752 tablet_service.cc:1468] Processing CreateTablet for tablet 6464fd2fa5bf46dabf3a177d5e756b94 (DEFAULT_TABLE table=default.Table_b9cb2b777214494e9a98a81d9b7785ee [id=9b580131b1d943d6b74bf75268435702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.122529 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6464fd2fa5bf46dabf3a177d5e756b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.122674 1159 tablet_bootstrap.cc:492] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.122761 1159 ts_tablet_manager.cc:1397] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.111s user 0.003s sys 0.000s
I20250629 01:52:01.123289 1159 raft_consensus.cc:357] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.123401 1159 raft_consensus.cc:383] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.123435 1159 raft_consensus.cc:738] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.078503 742 tablet_service.cc:1515] Processing DeleteTablet for tablet a014907d09c94745859436f1eb73b280 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.125928 2670 raft_consensus.cc:491] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.126029 2670 raft_consensus.cc:513] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.081244 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4dd23f6818f64b989dbb7aaaabcd816d (DEFAULT_TABLE table=default.Table_346dcc9d7d4c43fb8bc4573429d3aaa0 [id=be90c29ecc80438ead822197bf4f202c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.126706 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd23f6818f64b989dbb7aaaabcd816d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.088915 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8295e8178eff4d5eb95ac885092731e0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.128127 1159 consensus_queue.cc:260] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.128213 777 raft_consensus.cc:2466] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:01.094910 2649 raft_consensus.cc:491] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.128353 2649 raft_consensus.cc:513] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.128726 1159 ts_tablet_manager.cc:1428] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:52:01.128780 2649 leader_election.cc:290] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.095057 2684 tablet.cc:1722] T d456410cd09845dabd979c123f70741b P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:52:01.128918 1159 tablet_bootstrap.cc:492] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.129209 986 leader_election.cc:304] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.129544 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.129689 2527 raft_consensus.cc:2802] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.129746 2527 raft_consensus.cc:491] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.129801 2527 raft_consensus.cc:3058] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:01.129886 710 leader_election.cc:343] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.108301 1155 tablet_bootstrap.cc:654] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.112398 753 tablet_service.cc:1468] Processing CreateTablet for tablet b17c4768b35443cf8c289a96f2cf6d13 (DEFAULT_TABLE table=default.Table_26700fd18fbf4cddb81d599cf016884c [id=6e590d1c3cf7433280ae8c12f57fde40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.130124 1159 tablet_bootstrap.cc:654] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.130214 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b17c4768b35443cf8c289a96f2cf6d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.131862 2527 raft_consensus.cc:513] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.132223 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.132638 712 leader_election.cc:343] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.132719 712 leader_election.cc:304] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:01.133155 2670 leader_election.cc:290] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.133584 2649 raft_consensus.cc:2747] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.133889 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82a837932f14419c8b32fcd2c9358215" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.114405 740 tablet_service.cc:1468] Processing CreateTablet for tablet 2213f74c54534046a671808db21f6ee8 (DEFAULT_TABLE table=default.Table_073d8d6803d541acb4e3549a8ca2dca2 [id=161db0d2090f4f6faf8122fddd3942d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.135053 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2213f74c54534046a671808db21f6ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.136013 2526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1251) took 1116 ms. Trace:
I20250629 01:52:01.138834 2526 rpcz_store.cc:276] 0629 01:52:00.019937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.020168 (+ 231us) service_pool.cc:225] Handling call
0629 01:52:01.065096 (+1044928us) ts_tablet_manager.cc:1145] Deleting tablet 2845d8fbf3f34c769027b9b6bb3e1f08
0629 01:52:01.065105 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.105308 (+ 40203us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.107538 (+ 2230us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.135999 (+ 28461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1044783}
I20250629 01:52:01.140383 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 2845d8fbf3f34c769027b9b6bb3e1f08 (table default.Table_208f2046cc46452887e343bab242e4b2 [id=77c157c54c864ddbb4678e9af6b1fe43]) successfully deleted
I20250629 01:52:01.140789 2526 tablet_replica.cc:331] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.141034 2526 raft_consensus.cc:2241] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.141225 2526 raft_consensus.cc:2270] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.143088 2527 leader_election.cc:290] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.143515 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8295e8178eff4d5eb95ac885092731e0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:01.143919 988 leader_election.cc:343] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.144446 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8295e8178eff4d5eb95ac885092731e0" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:01.144568 777 raft_consensus.cc:3058] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.144657 2526 ts_tablet_manager.cc:1905] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.144644 1158 ts_tablet_manager.cc:1905] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.145028 2649 raft_consensus.cc:491] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.145114 2649 raft_consensus.cc:513] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.145522 2649 leader_election.cc:290] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.146155 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5d27d658892496a9b19b57949a8399a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.146474 712 leader_election.cc:343] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.146699 777 raft_consensus.cc:2466] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:01.146937 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5d27d658892496a9b19b57949a8399a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.147068 1051 raft_consensus.cc:2466] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:01.147364 710 leader_election.cc:304] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.147394 986 leader_election.cc:304] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.147679 2649 raft_consensus.cc:2802] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.147742 2649 raft_consensus.cc:491] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.147789 2649 raft_consensus.cc:3058] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.148063 2527 raft_consensus.cc:2802] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.148168 2527 raft_consensus.cc:695] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:01.148361 2527 consensus_queue.cc:237] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.149981 2649 raft_consensus.cc:513] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.150707 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5d27d658892496a9b19b57949a8399a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.150784 2649 leader_election.cc:290] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.150836 1051 raft_consensus.cc:3058] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.151194 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5d27d658892496a9b19b57949a8399a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:01.151528 712 leader_election.cc:343] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.152853 1051 raft_consensus.cc:2466] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:01.153303 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82a837932f14419c8b32fcd2c9358215" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.153424 710 leader_election.cc:304] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.153762 2649 raft_consensus.cc:2802] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.153731 850 leader_election.cc:304] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.153894 2649 raft_consensus.cc:695] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.154054 2670 raft_consensus.cc:2747] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.154111 2649 consensus_queue.cc:237] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.155476 1158 ts_tablet_manager.cc:1918] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.155581 1158 log.cc:1199] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ce90c045115844b0b96143f91f1be1d4
I20250629 01:52:01.155812 1158 ts_tablet_manager.cc:1939] T ce90c045115844b0b96143f91f1be1d4 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.155829 2526 ts_tablet_manager.cc:1918] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.155897 2526 log.cc:1199] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/2249d5cba1544653b088312e7d74802e
I20250629 01:52:01.156538 1156 log.cc:1199] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ce90c045115844b0b96143f91f1be1d4
I20250629 01:52:01.156742 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ce90c045115844b0b96143f91f1be1d4 (table default.Table_78b4b6aea1c144eeb3669ca5d081e0db [id=7cb4f3fa847544c0852c62f40a1c9ee3]) successfully deleted
I20250629 01:52:01.156814 1156 ts_tablet_manager.cc:1939] T ce90c045115844b0b96143f91f1be1d4 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:01.157451 1156 tablet_replica.cc:331] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:01.157593 1156 raft_consensus.cc:2241] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.157752 1156 pending_rounds.cc:70] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:01.157784 1156 pending_rounds.cc:77] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759225220898816 op_type: NO_OP noop_request { }
I20250629 01:52:01.157840 1156 raft_consensus.cc:2887] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.157892 1156 raft_consensus.cc:2270] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:01.158344 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 45ca747f03524076b4112e4db78614fe because the tablet was not found. No further retry: Not found: Tablet not found: 45ca747f03524076b4112e4db78614fe
I20250629 01:52:01.158720 1158 tablet_replica.cc:331] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.158854 1158 raft_consensus.cc:2241] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.158943 1158 raft_consensus.cc:2270] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.159191 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 0fc89468158e44a3bb604a590b07acf6 (DEFAULT_TABLE table=default.Table_fb78bf13072345fea20a096dffdba5fb [id=d88ad526d4344361b6081a2e8fc126e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.159457 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fc89468158e44a3bb604a590b07acf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.159545 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ce90c045115844b0b96143f91f1be1d4 (table default.Table_78b4b6aea1c144eeb3669ca5d081e0db [id=7cb4f3fa847544c0852c62f40a1c9ee3]) successfully deleted
I20250629 01:52:01.163204 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0fc89468158e44a3bb604a590b07acf6 (DEFAULT_TABLE table=default.Table_fb78bf13072345fea20a096dffdba5fb [id=d88ad526d4344361b6081a2e8fc126e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.163643 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fc89468158e44a3bb604a590b07acf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.164072 2526 ts_tablet_manager.cc:1939] T 2249d5cba1544653b088312e7d74802e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.164880 2526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1252) took 1130 ms. Trace:
I20250629 01:52:01.165261 1158 ts_tablet_manager.cc:1905] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:01.168530 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c46bd4bfa331420abdcd5fe729300fc4 consensus state has no leader
I20250629 01:52:01.168505 2526 rpcz_store.cc:276] 0629 01:52:00.033958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.034009 (+ 51us) service_pool.cc:225] Handling call
0629 01:52:01.140739 (+1106730us) ts_tablet_manager.cc:1145] Deleting tablet 2249d5cba1544653b088312e7d74802e
0629 01:52:01.140751 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.147070 (+ 6319us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.155765 (+ 8695us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.164864 (+ 9099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106589}
I20250629 01:52:01.168974 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2045012c80d049eaabf7dec1df28b087
I20250629 01:52:01.169139 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 2249d5cba1544653b088312e7d74802e (table default.Table_c1fb2548e78248e98b54bfb7f01c0e51 [id=a2b08c6d970a45709f099af55d84f522]) successfully deleted
I20250629 01:52:01.169436 2526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1253) took 1081 ms. Trace:
I20250629 01:52:01.169536 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 2045012c80d049eaabf7dec1df28b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.169502 2526 rpcz_store.cc:276] 0629 01:52:00.088151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.088199 (+ 48us) service_pool.cc:225] Handling call
0629 01:52:01.169352 (+1081153us) ts_tablet_manager.cc:1145] Deleting tablet e4ff12d702bf48039d63c437079979fd
0629 01:52:01.169360 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.169424 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1074331}
I20250629 01:52:01.169709 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 7edfdf771ecd4ae8b87791846536271f
I20250629 01:52:01.170100 454 catalog_manager.cc:6652] ReplaceTablet: tablet 7edfdf771ecd4ae8b87791846536271f deleted and replaced by tablet 229ab5b547634fb59089510905e49a3e
I20250629 01:52:01.170444 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edfdf771ecd4ae8b87791846536271f with delete_type TABLET_DATA_DELETED (replaced by tablet 229ab5b547634fb59089510905e49a3e) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.170465 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edfdf771ecd4ae8b87791846536271f with delete_type TABLET_DATA_DELETED (replaced by tablet 229ab5b547634fb59089510905e49a3e) from {username='slave'} at 127.0.0.1:38530
W20250629 01:52:01.170653 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e4ff12d702bf48039d63c437079979fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4ff12d702bf48039d63c437079979fd already in progress: creating tablet
W20250629 01:52:01.170818 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 24c4841e127b495cbfbca0103662c01e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24c4841e127b495cbfbca0103662c01e already in progress: creating tablet
I20250629 01:52:01.170878 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 2045012c80d049eaabf7dec1df28b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.171347 896 tablet_service.cc:1468] Processing CreateTablet for tablet f3db646a559340269980306633f26965 (DEFAULT_TABLE table=default.Table_cbb81a42b45040139dcda6493db2fe5f [id=1045a28ba2a54b6e9a969955a9d0f39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.171339 1023 tablet_service.cc:1468] Processing CreateTablet for tablet f3db646a559340269980306633f26965 (DEFAULT_TABLE table=default.Table_cbb81a42b45040139dcda6493db2fe5f [id=1045a28ba2a54b6e9a969955a9d0f39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.171646 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3db646a559340269980306633f26965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.172201 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7e5e094e14304fd7b0899ba75536ea4a" } new_table_name: "default.Table_d83a59629f3944a09f3598c45f9db645" modify_external_catalogs: true
I20250629 01:52:01.173530 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3db646a559340269980306633f26965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.173907 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6b4d4d6dacb84f80903522d20de4e795
I20250629 01:52:01.174280 448 catalog_manager.cc:6652] ReplaceTablet: tablet 6b4d4d6dacb84f80903522d20de4e795 deleted and replaced by tablet a6ace901512449f8b93281b8b1d69dde
I20250629 01:52:01.174541 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_75ece33d5b574d8b83f95340d391681f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.174702 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b4d4d6dacb84f80903522d20de4e795 with delete_type TABLET_DATA_DELETED (replaced by tablet a6ace901512449f8b93281b8b1d69dde) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.174765 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b4d4d6dacb84f80903522d20de4e795 with delete_type TABLET_DATA_DELETED (replaced by tablet a6ace901512449f8b93281b8b1d69dde) from {username='slave'} at 127.0.0.1:38530
W20250629 01:52:01.174968 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75ece33d5b574d8b83f95340d391681f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.175732 755 tablet_service.cc:1468] Processing CreateTablet for tablet f3db646a559340269980306633f26965 (DEFAULT_TABLE table=default.Table_cbb81a42b45040139dcda6493db2fe5f [id=1045a28ba2a54b6e9a969955a9d0f39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.176026 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3db646a559340269980306633f26965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.176541 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6b4d4d6dacb84f80903522d20de4e795 with delete_type TABLET_DATA_DELETED (replaced by tablet a6ace901512449f8b93281b8b1d69dde) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.176738 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edfdf771ecd4ae8b87791846536271f with delete_type TABLET_DATA_DELETED (replaced by tablet 229ab5b547634fb59089510905e49a3e) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.178005 2526 tablet_replica.cc:331] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.178669 2526 raft_consensus.cc:2241] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.178912 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 303f77a729c648f9b1146d11056b242d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.179214 2526 raft_consensus.cc:2270] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.179183 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5bbd9c09a3a2490ab18ec643b60078b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.179720 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbd9c09a3a2490ab18ec643b60078b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.179720 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.179824 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.179862 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 8295e8178eff4d5eb95ac885092731e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.180014 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.180387 2526 ts_tablet_manager.cc:1905] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:01.180985 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 27b76e04158e492a9979c06e7cc2ae37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27b76e04158e492a9979c06e7cc2ae37 already in progress: creating tablet
W20250629 01:52:01.181138 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4b11d32f1c3e41f3b9f6d20fd837226a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b11d32f1c3e41f3b9f6d20fd837226a already in progress: creating tablet
W20250629 01:52:01.181252 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9c66df5169224fea8c4fc45df7d695d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c66df5169224fea8c4fc45df7d695d8 already in progress: creating tablet
W20250629 01:52:01.181380 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet f93f07e50b414db29c081760ce2004c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f93f07e50b414db29c081760ce2004c1 already in progress: creating tablet
I20250629 01:52:01.181483 1158 ts_tablet_manager.cc:1918] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.179975 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_acd11ca3b08f4e69a5df56589bf21a9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.181564 1158 log.cc:1199] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/1fe0cd4b8f7d47268a80f645bc8b2b9d
I20250629 01:52:01.181774 1158 ts_tablet_manager.cc:1939] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:52:01.181799 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acd11ca3b08f4e69a5df56589bf21a9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.184356 1156 ts_tablet_manager.cc:1905] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.184296 896 tablet_service.cc:1468] Processing CreateTablet for tablet eab680c1f6f843e3a26b08c92fa95167 (DEFAULT_TABLE table=default.Table_238425ea966b4ee1b67de895973a1521 [id=759476ba85054d04b3c4d2644f530fd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.184648 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab680c1f6f843e3a26b08c92fa95167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.185072 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 82a837932f14419c8b32fcd2c9358215 (table default.Table_3592b5e7e0c44766a622a04a3a5c9b03 [id=ed3c8fa707924f8ea53d590a7f9b3d68]). Expected version 2 got 0
I20250629 01:52:01.185240 453 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9c799a04c0f74492b9fb674f9b0da240 (table default.Table_3c4a230556474a8e831d7180634f0155 [id=297fd4ccf3cb46c283189b67c1b8e648]). Expected version 2 got 0
I20250629 01:52:01.185286 1023 tablet_service.cc:1468] Processing CreateTablet for tablet eab680c1f6f843e3a26b08c92fa95167 (DEFAULT_TABLE table=default.Table_238425ea966b4ee1b67de895973a1521 [id=759476ba85054d04b3c4d2644f530fd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.185582 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab680c1f6f843e3a26b08c92fa95167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.187079 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 2045012c80d049eaabf7dec1df28b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.188351 740 tablet_service.cc:1468] Processing CreateTablet for tablet eab680c1f6f843e3a26b08c92fa95167 (DEFAULT_TABLE table=default.Table_238425ea966b4ee1b67de895973a1521 [id=759476ba85054d04b3c4d2644f530fd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.188660 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab680c1f6f843e3a26b08c92fa95167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.190279 896 tablet_service.cc:1468] Processing CreateTablet for tablet 594a47a6aac04d13ae13a11cb18263a4 (DEFAULT_TABLE table=default.Table_63d441b893a5478e83e7691f2e9dc343 [id=394e6fc2faf14689b494071799bb015e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.190605 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594a47a6aac04d13ae13a11cb18263a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.191162 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d (table default.Table_6b2a627b815e45c480dcaafe356a5d50 [id=d5875760fd97485b88f3c69340dc7b01]) successfully deleted
I20250629 01:52:01.191872 1158 tablet_replica.cc:331] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.192054 1158 raft_consensus.cc:2241] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.192529 1158 pending_rounds.cc:70] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:01.192764 1158 pending_rounds.cc:77] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759227408150528 op_type: NO_OP noop_request { }
I20250629 01:52:01.192934 1158 raft_consensus.cc:2887] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.193084 1158 raft_consensus.cc:2270] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.193884 1158 ts_tablet_manager.cc:1905] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.196683 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 594a47a6aac04d13ae13a11cb18263a4 (DEFAULT_TABLE table=default.Table_63d441b893a5478e83e7691f2e9dc343 [id=394e6fc2faf14689b494071799bb015e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.197063 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594a47a6aac04d13ae13a11cb18263a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.198217 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:01.198555 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:01.198678 451 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 76118a6e684448b09f244d44c909ccaa (table default.Table_5794ea9989b54ab9bd7ecca30428c971 [id=dcc07a46945a4add8e667bfea51dfed7]). Expected version 5 got 0
I20250629 01:52:01.200570 1155 tablet_bootstrap.cc:492] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.201685 1155 ts_tablet_manager.cc:1397] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.164s user 0.005s sys 0.000s
I20250629 01:52:01.201273 755 tablet_service.cc:1468] Processing CreateTablet for tablet 594a47a6aac04d13ae13a11cb18263a4 (DEFAULT_TABLE table=default.Table_63d441b893a5478e83e7691f2e9dc343 [id=394e6fc2faf14689b494071799bb015e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.202893 1155 raft_consensus.cc:357] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.203811 1159 tablet_bootstrap.cc:492] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.203944 1159 ts_tablet_manager.cc:1397] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.004s
I20250629 01:52:01.204232 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 867b59531961494bad28b0c442565948 (DEFAULT_TABLE table=default.Table_7788acac435247629f3a62bc49aa2ca0 [id=6f3de2df0b7341019a07d4b4a464ed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.204511 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 867b59531961494bad28b0c442565948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.204566 1159 raft_consensus.cc:357] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.204680 1159 raft_consensus.cc:383] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.204721 1159 raft_consensus.cc:738] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.204901 1159 consensus_queue.cc:260] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.207026 1159 ts_tablet_manager.cc:1428] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:01.207239 1159 tablet_bootstrap.cc:492] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.208339 1159 tablet_bootstrap.cc:654] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.208593 896 tablet_service.cc:1468] Processing CreateTablet for tablet 867b59531961494bad28b0c442565948 (DEFAULT_TABLE table=default.Table_7788acac435247629f3a62bc49aa2ca0 [id=6f3de2df0b7341019a07d4b4a464ed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.209342 1155 raft_consensus.cc:383] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.209437 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 867b59531961494bad28b0c442565948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.209637 1155 raft_consensus.cc:738] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.211195 1155 consensus_queue.cc:260] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.211694 1154 tablet_bootstrap.cc:492] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.211882 1154 ts_tablet_manager.cc:1397] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.172s user 0.007s sys 0.039s
I20250629 01:52:01.212618 1155 ts_tablet_manager.cc:1428] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250629 01:52:01.212930 1154 raft_consensus.cc:357] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.220584 2526 ts_tablet_manager.cc:1918] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.220712 2526 log.cc:1199] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8a880c9bb4b94d3b9b12f28d48565f06
I20250629 01:52:01.221036 2526 ts_tablet_manager.cc:1939] T 8a880c9bb4b94d3b9b12f28d48565f06 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.221483 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8a880c9bb4b94d3b9b12f28d48565f06 (table default.Table_0ed9b0a279de49078207546c6ddbc21f [id=4457fae28052479e81efac2f717c1bb6]) successfully deleted
W20250629 01:52:01.222012 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet b5bcfb2ac1674beb8e9d50d2818cec0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bcfb2ac1674beb8e9d50d2818cec0d already in progress: creating tablet
I20250629 01:52:01.222121 1155 tablet_bootstrap.cc:492] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:01.222517 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e3874404c3444d47a71f1dce7fb86ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3874404c3444d47a71f1dce7fb86ff8 already in progress: creating tablet
W20250629 01:52:01.222997 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 8ae366e6ccf34a9faf2caa2bded63546 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae366e6ccf34a9faf2caa2bded63546 already in progress: creating tablet
I20250629 01:52:01.223317 1154 raft_consensus.cc:383] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.223418 1154 raft_consensus.cc:738] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.223628 1154 consensus_queue.cc:260] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.223868 1155 tablet_bootstrap.cc:654] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:01.223901 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet bae91d28f27f4e14a23c65e41091b671 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bae91d28f27f4e14a23c65e41091b671 already in progress: creating tablet
W20250629 01:52:01.224484 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c2591c65d53449c8801778606f1f05f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2591c65d53449c8801778606f1f05f1 already in progress: creating tablet
I20250629 01:52:01.226212 2526 tablet_replica.cc:331] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.226451 2527 raft_consensus.cc:491] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.226567 2526 raft_consensus.cc:2241] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.226595 2527 raft_consensus.cc:513] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.226917 2526 raft_consensus.cc:2270] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.227520 2526 ts_tablet_manager.cc:1905] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.227496 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "327c303b6b9347f5b4cb78f74d41a44a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.227548 2527 leader_election.cc:290] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.227627 777 raft_consensus.cc:2466] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:01.227994 986 leader_election.cc:304] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.228160 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "327c303b6b9347f5b4cb78f74d41a44a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.228227 2527 raft_consensus.cc:2802] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.228307 2527 raft_consensus.cc:491] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.228366 2527 raft_consensus.cc:3058] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:01.228658 988 leader_election.cc:343] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.230618 2527 raft_consensus.cc:513] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.231292 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "327c303b6b9347f5b4cb78f74d41a44a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:01.231415 777 raft_consensus.cc:3058] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.231457 2527 leader_election.cc:290] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.231750 1154 ts_tablet_manager.cc:1428] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250629 01:52:01.231947 1154 tablet_bootstrap.cc:492] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.233585 777 raft_consensus.cc:2466] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:01.233918 986 leader_election.cc:304] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.234071 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "327c303b6b9347f5b4cb78f74d41a44a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.234120 2527 raft_consensus.cc:2802] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.234202 2527 raft_consensus.cc:695] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:01.234390 2527 consensus_queue.cc:237] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:01.236428 988 leader_election.cc:343] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.237285 915 raft_consensus.cc:3058] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.241161 1154 tablet_bootstrap.cc:654] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.242156 2526 ts_tablet_manager.cc:1918] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.242957 1159 tablet_bootstrap.cc:492] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.243412 1159 ts_tablet_manager.cc:1397] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.002s
I20250629 01:52:01.243995 1159 raft_consensus.cc:357] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.244185 1159 raft_consensus.cc:383] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.244264 1159 raft_consensus.cc:738] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.244472 1159 consensus_queue.cc:260] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.244917 1159 ts_tablet_manager.cc:1428] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:01.245144 1159 tablet_bootstrap.cc:492] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.246277 1159 tablet_bootstrap.cc:654] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.246567 740 tablet_service.cc:1468] Processing CreateTablet for tablet 867b59531961494bad28b0c442565948 (DEFAULT_TABLE table=default.Table_7788acac435247629f3a62bc49aa2ca0 [id=6f3de2df0b7341019a07d4b4a464ed1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.246811 2526 log.cc:1199] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/adbb648c49e24b8085eacbe6e0a86991
I20250629 01:52:01.247092 2526 ts_tablet_manager.cc:1939] T adbb648c49e24b8085eacbe6e0a86991 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.247491 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet adbb648c49e24b8085eacbe6e0a86991 (table default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]) successfully deleted
W20250629 01:52:01.248104 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a014907d09c94745859436f1eb73b280 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a014907d09c94745859436f1eb73b280 already in progress: creating tablet
I20250629 01:52:01.248392 896 tablet_service.cc:1468] Processing CreateTablet for tablet 7374fec0a237447faf2cddd9b891a643 (DEFAULT_TABLE table=default.Table_1fcaf9bb8e77453f820db3cd77028eaf [id=a1564e0519a841869e62de376aab62ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.248497 2487 tablet.cc:1722] T 8f9f59fbe9634ac8b27f990d9496f637 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:01.248673 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7374fec0a237447faf2cddd9b891a643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.249472 1158 ts_tablet_manager.cc:1918] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.249543 1158 log.cc:1199] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e4d6f1eb43b24fda94aa0afcf4501195
I20250629 01:52:01.250681 1158 ts_tablet_manager.cc:1939] T e4d6f1eb43b24fda94aa0afcf4501195 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.251111 2649 raft_consensus.cc:491] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.251207 2649 raft_consensus.cc:513] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.251436 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e4d6f1eb43b24fda94aa0afcf4501195 (table default.Table_1e578e5fe38c40d38ba9b7759798a267 [id=64e6f3a1044a485dac278b2deccc979d]) successfully deleted
I20250629 01:52:01.251749 1158 tablet_replica.cc:331] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.251883 1158 raft_consensus.cc:2241] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.251924 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ab0aa4e74d345b2a214b90b5018c82b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.251978 1158 raft_consensus.cc:2270] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:01.252323 712 leader_election.cc:343] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.252482 1158 ts_tablet_manager.cc:1905] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.253319 2649 leader_election.cc:290] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.253752 1154 tablet_bootstrap.cc:492] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.253842 1154 ts_tablet_manager.cc:1397] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250629 01:52:01.254390 1154 raft_consensus.cc:357] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.254518 1154 raft_consensus.cc:383] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.254556 1154 raft_consensus.cc:738] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.254928 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ab0aa4e74d345b2a214b90b5018c82b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.255064 1048 raft_consensus.cc:2466] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:01.255403 710 leader_election.cc:304] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.255836 2049 raft_consensus.cc:2802] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.255903 2049 raft_consensus.cc:491] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.255957 2049 raft_consensus.cc:3058] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.257341 1158 ts_tablet_manager.cc:1918] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.257414 1158 log.cc:1199] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f94d8fc1b6254ba69acadcaff3613d97
I20250629 01:52:01.257630 1158 ts_tablet_manager.cc:1939] T f94d8fc1b6254ba69acadcaff3613d97 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.258095 2049 raft_consensus.cc:513] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.258337 1159 tablet_bootstrap.cc:492] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.258428 1159 ts_tablet_manager.cc:1397] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250629 01:52:01.258562 2049 leader_election.cc:290] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.258776 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ab0aa4e74d345b2a214b90b5018c82b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.258905 1159 raft_consensus.cc:357] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.259022 1159 raft_consensus.cc:383] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.259063 1159 raft_consensus.cc:738] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:52:01.259122 712 leader_election.cc:343] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.259233 1159 consensus_queue.cc:260] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.259691 1159 ts_tablet_manager.cc:1428] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:01.259882 1159 tablet_bootstrap.cc:492] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.260561 892 tablet_service.cc:1468] Processing CreateTablet for tablet d6ced5759a50401a9d6c4bf075d94732 (DEFAULT_TABLE table=default.Table_1d228257fd614e23b098ddd7c3e7d0d8 [id=43d4ba978eae44e4a0375780d73c9fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.260816 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ced5759a50401a9d6c4bf075d94732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.261317 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6ccf1819ca3f4059a95b5cda8096c2dd (DEFAULT_TABLE table=default.Table_c5371085a4334fbfbd6f4e385058da15 [id=c9ae373ac0c1490b9fcfc0081c40753f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.261386 1159 tablet_bootstrap.cc:654] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.261581 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ccf1819ca3f4059a95b5cda8096c2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.262044 1154 consensus_queue.cc:260] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.262997 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:01.263307 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 76118a6e684448b09f244d44c909ccaa (table default.Table_5794ea9989b54ab9bd7ecca30428c971 [id=dcc07a46945a4add8e667bfea51dfed7]). Expected version 5 got 0
I20250629 01:52:01.265009 757 tablet_service.cc:1468] Processing CreateTablet for tablet d6ced5759a50401a9d6c4bf075d94732 (DEFAULT_TABLE table=default.Table_1d228257fd614e23b098ddd7c3e7d0d8 [id=43d4ba978eae44e4a0375780d73c9fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.265233 752 tablet_service.cc:1468] Processing CreateTablet for tablet 6ccf1819ca3f4059a95b5cda8096c2dd (DEFAULT_TABLE table=default.Table_c5371085a4334fbfbd6f4e385058da15 [id=c9ae373ac0c1490b9fcfc0081c40753f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:01.265415 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2045012c80d049eaabf7dec1df28b087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2045012c80d049eaabf7dec1df28b087 already in progress: creating tablet
W20250629 01:52:01.265589 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 6b4d4d6dacb84f80903522d20de4e795 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4d4d6dacb84f80903522d20de4e795 already in progress: creating tablet
W20250629 01:52:01.265722 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 7edfdf771ecd4ae8b87791846536271f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7edfdf771ecd4ae8b87791846536271f already in progress: creating tablet
I20250629 01:52:01.265785 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet f94d8fc1b6254ba69acadcaff3613d97 (table default.Table_3fc2c8b5061f4a6e8f46cd439cf5c776 [id=92d55fc2cf6343d7aac796eb0ddb93f8]) successfully deleted
I20250629 01:52:01.265934 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 7374fec0a237447faf2cddd9b891a643 (DEFAULT_TABLE table=default.Table_1fcaf9bb8e77453f820db3cd77028eaf [id=a1564e0519a841869e62de376aab62ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.265981 877 tablet_service.cc:1468] Processing CreateTablet for tablet 0cece970f6ce485dbf3a30d6ebf3a474 (DEFAULT_TABLE table=default.Table_3181bf94172b4d6f95360a4c1be43602 [id=9b85ddf6a065467da0233816102483ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.266206 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7374fec0a237447faf2cddd9b891a643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.266186 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ab0aa4e74d345b2a214b90b5018c82b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.266304 1048 raft_consensus.cc:3058] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.267174 753 tablet_service.cc:1468] Processing CreateTablet for tablet 7374fec0a237447faf2cddd9b891a643 (DEFAULT_TABLE table=default.Table_1fcaf9bb8e77453f820db3cd77028eaf [id=a1564e0519a841869e62de376aab62ba]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:01.267222 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2045012c80d049eaabf7dec1df28b087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2045012c80d049eaabf7dec1df28b087 already in progress: creating tablet
W20250629 01:52:01.267704 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 7edfdf771ecd4ae8b87791846536271f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7edfdf771ecd4ae8b87791846536271f already in progress: creating tablet
W20250629 01:52:01.268191 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6b4d4d6dacb84f80903522d20de4e795 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4d4d6dacb84f80903522d20de4e795 already in progress: creating tablet
I20250629 01:52:01.268322 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 6ccf1819ca3f4059a95b5cda8096c2dd (DEFAULT_TABLE table=default.Table_c5371085a4334fbfbd6f4e385058da15 [id=c9ae373ac0c1490b9fcfc0081c40753f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.268510 1048 raft_consensus.cc:2466] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:01.268584 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ccf1819ca3f4059a95b5cda8096c2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.268826 710 leader_election.cc:304] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.269184 2049 raft_consensus.cc:2802] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.269385 2049 raft_consensus.cc:695] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.269750 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ced5759a50401a9d6c4bf075d94732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.269919 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ccf1819ca3f4059a95b5cda8096c2dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.270143 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594a47a6aac04d13ae13a11cb18263a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.269615 2049 consensus_queue.cc:237] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.270457 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 867b59531961494bad28b0c442565948. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.272286 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d6ced5759a50401a9d6c4bf075d94732 (DEFAULT_TABLE table=default.Table_1d228257fd614e23b098ddd7c3e7d0d8 [id=43d4ba978eae44e4a0375780d73c9fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.272521 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ced5759a50401a9d6c4bf075d94732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.273427 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7374fec0a237447faf2cddd9b891a643. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.279376 1159 tablet_bootstrap.cc:492] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.279521 1159 ts_tablet_manager.cc:1397] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250629 01:52:01.280200 1159 raft_consensus.cc:357] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.280548 1159 raft_consensus.cc:383] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.280632 1159 raft_consensus.cc:738] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.280838 1159 consensus_queue.cc:260] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.281307 1159 ts_tablet_manager.cc:1428] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.281554 1159 tablet_bootstrap.cc:492] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.282618 1156 ts_tablet_manager.cc:1918] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.282755 1156 log.cc:1199] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/1fe0cd4b8f7d47268a80f645bc8b2b9d
W20250629 01:52:01.282913 712 consensus_peers.cc:489] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:01.282967 1159 tablet_bootstrap.cc:654] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.283186 1156 ts_tablet_manager.cc:1939] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:01.283973 1156 tablet_replica.cc:331] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:01.284349 1156 raft_consensus.cc:2241] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.284399 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d (table default.Table_6b2a627b815e45c480dcaafe356a5d50 [id=d5875760fd97485b88f3c69340dc7b01]) successfully deleted
I20250629 01:52:01.284500 1156 raft_consensus.cc:2270] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.284865 1156 ts_tablet_manager.cc:1905] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.286906 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7bc2154f17064cccaeadeb7ed80a3828 (DEFAULT_TABLE table=default.Table_94f201340df34d00a02cde31f001d6d7 [id=3486ac514a5b40548dd9c6b6d23a6fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.286898 1017 tablet_service.cc:1468] Processing CreateTablet for tablet dd7a6028e372422e88079618e748a2bf (DEFAULT_TABLE table=default.Table_4a3390362d0b4fa3ac770487937283b7 [id=f51d779a23d84260b34245d2a9634e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.287174 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7a6028e372422e88079618e748a2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.287937 884 tablet_service.cc:1468] Processing CreateTablet for tablet ad9ff2e769b0421299f6b96ebcecfb70 (DEFAULT_TABLE table=default.Table_ad8669ae4f784da9b23f7890936e487d [id=bd15b845d5614811b6255c6964a5d36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.288941 880 tablet_service.cc:1468] Processing CreateTablet for tablet d0ce16e94c504423b8061238a0405fea (DEFAULT_TABLE table=default.Table_2a4864f0d4da45b7b83ae578092fc4d7 [id=dec1d1c3a697410e861d4836cf3b95a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.289995 892 tablet_service.cc:1468] Processing CreateTablet for tablet dd7a6028e372422e88079618e748a2bf (DEFAULT_TABLE table=default.Table_4a3390362d0b4fa3ac770487937283b7 [id=f51d779a23d84260b34245d2a9634e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.296689 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 20ca3db3bedd4ce8bd8159512a15a3e6
I20250629 01:52:01.297354 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a792a26725954adea15ad57e31bfb224
I20250629 01:52:01.297696 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 20ca3db3bedd4ce8bd8159512a15a3e6 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:01.297964 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a1ddbbbe43044a70a80e7dd66011de81
W20250629 01:52:01.298576 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72a9ca562aa8445e9b26b789d1ebc02b consensus state has no leader
I20250629 01:52:01.302486 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.302678 886 tablet_service.cc:1515] Processing DeleteTablet for tablet a792a26725954adea15ad57e31bfb224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.302815 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a1ddbbbe43044a70a80e7dd66011de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.302842 2526 tablet_replica.cc:331] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.303072 2526 raft_consensus.cc:2241] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.303215 2526 raft_consensus.cc:2270] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:01.303529 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20ca3db3bedd4ce8bd8159512a15a3e6 already in progress: creating tablet
I20250629 01:52:01.303689 2526 ts_tablet_manager.cc:1905] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.308283 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cece970f6ce485dbf3a30d6ebf3a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.308449 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ce16e94c504423b8061238a0405fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.308600 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bc2154f17064cccaeadeb7ed80a3828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.308919 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9ff2e769b0421299f6b96ebcecfb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.309262 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7a6028e372422e88079618e748a2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.312062 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu pid 1851
I20250629 01:52:01.323699 2526 ts_tablet_manager.cc:1918] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:01.324155 2526 log.cc:1199] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a792a26725954adea15ad57e31bfb224
I20250629 01:52:01.324522 2526 ts_tablet_manager.cc:1939] T a792a26725954adea15ad57e31bfb224 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.324909 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]) successfully deleted
W20250629 01:52:01.325160 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a1ddbbbe43044a70a80e7dd66011de81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ddbbbe43044a70a80e7dd66011de81 already in progress: creating tablet
I20250629 01:52:01.325961 1159 tablet_bootstrap.cc:492] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.326100 1159 ts_tablet_manager.cc:1397] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.000s
I20250629 01:52:01.326843 1159 raft_consensus.cc:357] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.327028 1159 raft_consensus.cc:383] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.327105 1159 raft_consensus.cc:738] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.327438 1159 consensus_queue.cc:260] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.327946 1159 ts_tablet_manager.cc:1428] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.328213 1159 tablet_bootstrap.cc:492] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.329658 1159 tablet_bootstrap.cc:654] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.341272 1159 tablet_bootstrap.cc:492] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.341414 1159 ts_tablet_manager.cc:1397] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250629 01:52:01.342056 1159 raft_consensus.cc:357] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.342228 1159 raft_consensus.cc:383] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.342316 1159 raft_consensus.cc:738] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.342497 1159 consensus_queue.cc:260] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.342687 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1fac9c587cca42ab8f027b8dc01a79f3 (DEFAULT_TABLE table=default.Table_49b84436278348afa56cf408cab52a43 [id=c3679a71719945e79b6238e35882d871]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.343017 1159 ts_tablet_manager.cc:1428] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.343134 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fac9c587cca42ab8f027b8dc01a79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.343317 1159 tablet_bootstrap.cc:492] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.344590 1159 tablet_bootstrap.cc:654] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.352332 1017 tablet_service.cc:1468] Processing CreateTablet for tablet ad9ff2e769b0421299f6b96ebcecfb70 (DEFAULT_TABLE table=default.Table_ad8669ae4f784da9b23f7890936e487d [id=bd15b845d5614811b6255c6964a5d36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.352664 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9ff2e769b0421299f6b96ebcecfb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.355556 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 0cece970f6ce485dbf3a30d6ebf3a474 (DEFAULT_TABLE table=default.Table_3181bf94172b4d6f95360a4c1be43602 [id=9b85ddf6a065467da0233816102483ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.355835 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cece970f6ce485dbf3a30d6ebf3a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.356472 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet a1ddbbbe43044a70a80e7dd66011de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.356563 1154 ts_tablet_manager.cc:1428] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.103s user 0.001s sys 0.000s
I20250629 01:52:01.356916 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7bc2154f17064cccaeadeb7ed80a3828 (DEFAULT_TABLE table=default.Table_94f201340df34d00a02cde31f001d6d7 [id=3486ac514a5b40548dd9c6b6d23a6fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.357061 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 1fac9c587cca42ab8f027b8dc01a79f3 (DEFAULT_TABLE table=default.Table_49b84436278348afa56cf408cab52a43 [id=c3679a71719945e79b6238e35882d871]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.357152 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.357197 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bc2154f17064cccaeadeb7ed80a3828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.357484 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fac9c587cca42ab8f027b8dc01a79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.357860 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet a792a26725954adea15ad57e31bfb224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.358117 1031 tablet_service.cc:1468] Processing CreateTablet for tablet d0ce16e94c504423b8061238a0405fea (DEFAULT_TABLE table=default.Table_2a4864f0d4da45b7b83ae578092fc4d7 [id=dec1d1c3a697410e861d4836cf3b95a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.358343 1158 tablet_replica.cc:331] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.358404 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ce16e94c504423b8061238a0405fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.358487 1158 raft_consensus.cc:2241] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.358630 1158 raft_consensus.cc:2270] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.359076 1158 ts_tablet_manager.cc:1905] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:01.359426 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a1ddbbbe43044a70a80e7dd66011de81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ddbbbe43044a70a80e7dd66011de81 already in progress: creating tablet
W20250629 01:52:01.359623 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20ca3db3bedd4ce8bd8159512a15a3e6 already in progress: creating tablet
I20250629 01:52:01.365744 1154 tablet_bootstrap.cc:492] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.366117 1158 ts_tablet_manager.cc:1918] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:01.366196 1158 log.cc:1199] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a792a26725954adea15ad57e31bfb224
I20250629 01:52:01.366528 1158 ts_tablet_manager.cc:1939] T a792a26725954adea15ad57e31bfb224 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.367120 1154 tablet_bootstrap.cc:654] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:01.368026 2476 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 15565446718848418067) tablet 00000000000000000000000000000000 cur holder 0x61700008f500
I20250629 01:52:01.372074 2670 raft_consensus.cc:491] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.372210 2670 raft_consensus.cc:513] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.372677 2670 leader_election.cc:290] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:01.376441 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.380216 2527 consensus_queue.cc:1035] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:01.381320 915 raft_consensus.cc:3058] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.382164 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]) successfully deleted
I20250629 01:52:01.384058 2527 raft_consensus.cc:491] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.384204 2527 raft_consensus.cc:513] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.384789 2527 leader_election.cc:290] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.389189 1154 tablet_bootstrap.cc:492] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.389338 1154 ts_tablet_manager.cc:1397] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20250629 01:52:01.389979 1154 raft_consensus.cc:357] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.390157 1154 raft_consensus.cc:383] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.390244 1154 raft_consensus.cc:738] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.390468 1154 consensus_queue.cc:260] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.390925 1154 ts_tablet_manager.cc:1428] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:01.391180 1154 tablet_bootstrap.cc:492] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.393023 1154 tablet_bootstrap.cc:654] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.397967 912 raft_consensus.cc:3058] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.398931 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7908afa2abaa4521b44f954e0790f986" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.399727 988 leader_election.cc:343] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.409612 1154 tablet_bootstrap.cc:492] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.409870 1154 ts_tablet_manager.cc:1397] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20250629 01:52:01.410451 1154 raft_consensus.cc:357] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.410636 1154 raft_consensus.cc:383] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.410720 1154 raft_consensus.cc:738] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.410928 1154 consensus_queue.cc:260] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.411386 1154 ts_tablet_manager.cc:1428] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:01.411618 1154 tablet_bootstrap.cc:492] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.412932 2580 raft_consensus.cc:491] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.413028 2580 raft_consensus.cc:513] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.404641 911 raft_consensus.cc:3058] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.413476 2580 leader_election.cc:290] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.413844 1154 tablet_bootstrap.cc:654] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.413683 1159 tablet_bootstrap.cc:492] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.414003 1159 ts_tablet_manager.cc:1397] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.003s
I20250629 01:52:01.414710 1159 raft_consensus.cc:357] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.414901 1159 raft_consensus.cc:383] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.414975 1159 raft_consensus.cc:738] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.415189 1159 consensus_queue.cc:260] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.415745 1159 ts_tablet_manager.cc:1428] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:01.416132 1159 tablet_bootstrap.cc:492] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.413817 913 raft_consensus.cc:3058] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.417946 1159 tablet_bootstrap.cc:654] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.419570 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecbfc4e45dc9435bbd7a55ba7b7e4155" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.419904 988 leader_election.cc:343] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.420193 1154 tablet_bootstrap.cc:492] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.420337 1154 ts_tablet_manager.cc:1397] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250629 01:52:01.420939 1154 raft_consensus.cc:357] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.421355 1154 raft_consensus.cc:383] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.421455 1154 raft_consensus.cc:738] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.421694 1154 consensus_queue.cc:260] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.422135 1154 ts_tablet_manager.cc:1428] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.439646 1154 tablet_bootstrap.cc:492] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.441546 2487 tablet.cc:1722] T 82a837932f14419c8b32fcd2c9358215 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:01.449903 1154 tablet_bootstrap.cc:654] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.451846 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_e2f8e877756d4bf1a9535269e5e28bfc [id=15f7a887c8224335aaf55ad0469bbf12] alter complete (version 2)
I20250629 01:52:01.286196 742 tablet_service.cc:1468] Processing CreateTablet for tablet dd7a6028e372422e88079618e748a2bf (DEFAULT_TABLE table=default.Table_4a3390362d0b4fa3ac770487937283b7 [id=f51d779a23d84260b34245d2a9634e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.452680 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7a6028e372422e88079618e748a2bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.453122 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_21ecde86214e4cf4a564659ff667e485" } new_table_name: "default.Table_251b2f2e0e534894863348eb187eb363" modify_external_catalogs: true
I20250629 01:52:01.453440 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_e7206ec77eb44e729b77a7bda6dea373 [id=4525666163d64ba981dfff1c095c6ee2] alter complete (version 2)
I20250629 01:52:01.453996 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0c6f823d48f243c3809066323ee8deae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.454553 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c6f823d48f243c3809066323ee8deae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.454880 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_12dfc35bb7ba4e1792cce572411cdeb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.455248 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12dfc35bb7ba4e1792cce572411cdeb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.455497 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_69c6504c160847438c542810417992c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.455827 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69c6504c160847438c542810417992c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.455930 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_19d55c4c3f444c41b10920604792dccb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.287642 751 tablet_service.cc:1468] Processing CreateTablet for tablet 7bc2154f17064cccaeadeb7ed80a3828 (DEFAULT_TABLE table=default.Table_94f201340df34d00a02cde31f001d6d7 [id=3486ac514a5b40548dd9c6b6d23a6fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.456346 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bc2154f17064cccaeadeb7ed80a3828. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.457571 1156 ts_tablet_manager.cc:1918] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.457667 1156 log.cc:1199] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/1b0782c7e652419096710563930f2072
I20250629 01:52:01.457931 1156 ts_tablet_manager.cc:1939] T 1b0782c7e652419096710563930f2072 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:01.459343 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5d389e95bd2c44a28cbcc84d1d47a30f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.459667 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d389e95bd2c44a28cbcc84d1d47a30f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.288651 750 tablet_service.cc:1468] Processing CreateTablet for tablet ad9ff2e769b0421299f6b96ebcecfb70 (DEFAULT_TABLE table=default.Table_ad8669ae4f784da9b23f7890936e487d [id=bd15b845d5614811b6255c6964a5d36a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.460487 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad9ff2e769b0421299f6b96ebcecfb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.460563 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_910f8f6757c44909a04eb629bfa037e8" } new_table_name: "default.Table_641599ef5c234351b696b05dad91d8d8" modify_external_catalogs: true
I20250629 01:52:01.460661 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8a62929d23e2422e979440dd100634a5" } modify_external_catalogs: true
I20250629 01:52:01.460753 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8a62929d23e2422e979440dd100634a5" } modify_external_catalogs: true
I20250629 01:52:01.462402 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2d84c16e159a46469c09eba21ca1f592"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.462762 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d84c16e159a46469c09eba21ca1f592 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.289750 740 tablet_service.cc:1468] Processing CreateTablet for tablet 0cece970f6ce485dbf3a30d6ebf3a474 (DEFAULT_TABLE table=default.Table_3181bf94172b4d6f95360a4c1be43602 [id=9b85ddf6a065467da0233816102483ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.463812 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cece970f6ce485dbf3a30d6ebf3a474. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.465876 896 tablet_service.cc:1515] Processing DeleteTablet for tablet cb7314fbdfc74b1c8d9d1be9c05244cb with delete_type TABLET_DATA_DELETED (replaced by tablet 9c1dc6d6d2b1458cb57875d2e25354bc) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.466063 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 20586ba2af9e4cb7adec5ca6c3b76e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.466163 2526 tablet_replica.cc:331] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.466190 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 9cfff25bea764e62954e49611b71703c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.466325 2526 raft_consensus.cc:2241] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.466459 2526 raft_consensus.cc:2270] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.466907 2526 ts_tablet_manager.cc:1905] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.305783 757 tablet_service.cc:1515] Processing DeleteTablet for tablet a792a26725954adea15ad57e31bfb224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.305917 755 tablet_service.cc:1515] Processing DeleteTablet for tablet a1ddbbbe43044a70a80e7dd66011de81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.306010 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.332899 1155 tablet_bootstrap.cc:492] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.468003 1155 ts_tablet_manager.cc:1397] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.246s user 0.000s sys 0.003s
I20250629 01:52:01.468808 1155 raft_consensus.cc:357] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.468989 1155 raft_consensus.cc:383] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.469058 1155 raft_consensus.cc:738] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.469246 1155 consensus_queue.cc:260] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.469684 1155 ts_tablet_manager.cc:1428] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.469916 1155 tablet_bootstrap.cc:492] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.471203 1155 tablet_bootstrap.cc:654] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.338060 753 tablet_service.cc:1468] Processing CreateTablet for tablet d0ce16e94c504423b8061238a0405fea (DEFAULT_TABLE table=default.Table_2a4864f0d4da45b7b83ae578092fc4d7 [id=dec1d1c3a697410e861d4836cf3b95a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.472074 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0ce16e94c504423b8061238a0405fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.473090 1154 tablet_bootstrap.cc:492] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.473183 1154 ts_tablet_manager.cc:1397] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20250629 01:52:01.473896 1154 raft_consensus.cc:357] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.474041 1154 raft_consensus.cc:383] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.474093 1154 raft_consensus.cc:738] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.474242 1154 consensus_queue.cc:260] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.474642 1154 ts_tablet_manager.cc:1428] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:01.474865 1154 tablet_bootstrap.cc:492] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.475733 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.475904 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.476039 1158 tablet_replica.cc:331] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.476055 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 50d23f589de84dd5b150a17cee00c411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.476222 1158 raft_consensus.cc:2241] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.476325 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.476392 1158 pending_rounds.cc:70] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:01.476418 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 303f77a729c648f9b1146d11056b242d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.476449 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.348407 747 tablet_service.cc:1468] Processing CreateTablet for tablet 1fac9c587cca42ab8f027b8dc01a79f3 (DEFAULT_TABLE table=default.Table_49b84436278348afa56cf408cab52a43 [id=c3679a71719945e79b6238e35882d871]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.476568 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.476600 2527 raft_consensus.cc:491] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.476698 2527 raft_consensus.cc:513] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.476787 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fac9c587cca42ab8f027b8dc01a79f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.477243 2527 leader_election.cc:290] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:01.478143 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19d55c4c3f444c41b10920604792dccb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:01.479238 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e4d6f1eb43b24fda94aa0afcf4501195 because the tablet was not found. No further retry: Not found: Tablet not found: e4d6f1eb43b24fda94aa0afcf4501195
I20250629 01:52:01.374507 2649 consensus_queue.cc:1035] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:01.376658 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c799a04c0f74492b9fb674f9b0da240" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.388311 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7908afa2abaa4521b44f954e0790f986" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.400400 2651 raft_consensus.cc:491] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.480316 2651 raft_consensus.cc:513] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.480774 2651 leader_election.cc:290] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.404490 2534 raft_consensus.cc:491] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.476428 1158 pending_rounds.cc:77] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759229393473536 op_type: NO_OP noop_request { }
I20250629 01:52:01.480922 2534 raft_consensus.cc:513] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.481000 1158 raft_consensus.cc:2887] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.481061 1158 raft_consensus.cc:2270] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.481346 2534 leader_election.cc:290] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:01.408226 712 consensus_peers.cc:489] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:01.482195 1158 ts_tablet_manager.cc:1905] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.482241 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.482431 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 8295e8178eff4d5eb95ac885092731e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:01.440150 710 consensus_peers.cc:489] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a792a26725954adea15ad57e31bfb224. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:01.484169 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 44058.6us
I20250629 01:52:01.485148 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.485446 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76118a6e684448b09f244d44c909ccaa" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:01.485544 710 leader_election.cc:343] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.485579 1051 raft_consensus.cc:2466] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:01.487028 1158 ts_tablet_manager.cc:1918] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.487097 1158 log.cc:1199] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:52:01.487347 1158 ts_tablet_manager.cc:1939] T 327c303b6b9347f5b4cb78f74d41a44a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.488616 710 leader_election.cc:304] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.488843 2534 raft_consensus.cc:2802] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.488904 2534 raft_consensus.cc:491] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.488961 2534 raft_consensus.cc:3058] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.490996 2534 raft_consensus.cc:513] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.491492 2534 leader_election.cc:290] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.491997 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76118a6e684448b09f244d44c909ccaa" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.492151 1051 raft_consensus.cc:3058] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.494891 2580 raft_consensus.cc:491] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.494992 2580 raft_consensus.cc:513] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.495535 2580 leader_election.cc:290] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.495898 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e0727102d54294a7c13a4aeef311e1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.496140 1051 raft_consensus.cc:2466] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:01.496346 988 leader_election.cc:343] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.496510 710 leader_election.cc:304] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.498965 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 327c303b6b9347f5b4cb78f74d41a44a (table default.Table_209b7bc3f1de4919a043f2ab3c1c5d6a [id=d62355a430724b96abb294c451c9bd58]) successfully deleted
I20250629 01:52:01.499246 1158 tablet_replica.cc:331] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.499401 1158 raft_consensus.cc:2241] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.499508 1158 raft_consensus.cc:2270] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.499900 1158 ts_tablet_manager.cc:1905] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.500166 2534 raft_consensus.cc:2802] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.500346 2534 raft_consensus.cc:695] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.500525 2534 consensus_queue.cc:237] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.501188 1154 tablet_bootstrap.cc:654] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.504746 1155 tablet_bootstrap.cc:492] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.504894 1155 ts_tablet_manager.cc:1397] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20250629 01:52:01.505502 1155 raft_consensus.cc:357] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.505671 1155 raft_consensus.cc:383] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.505738 1155 raft_consensus.cc:738] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.505899 1155 consensus_queue.cc:260] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.506289 1155 ts_tablet_manager.cc:1428] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:52:01.506516 1155 tablet_bootstrap.cc:492] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.507586 1155 tablet_bootstrap.cc:654] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:01.511451 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 103244us
I20250629 01:52:01.511714 850 leader_election.cc:304] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:01.512081 986 leader_election.cc:343] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.512346 986 leader_election.cc:304] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.512512 2580 raft_consensus.cc:2747] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.516261 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76118a6e684448b09f244d44c909ccaa" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.516777 988 leader_election.cc:343] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.525727 2526 ts_tablet_manager.cc:1918] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.525969 2526 log.cc:1199] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/cb7314fbdfc74b1c8d9d1be9c05244cb
I20250629 01:52:01.526314 2526 ts_tablet_manager.cc:1939] T cb7314fbdfc74b1c8d9d1be9c05244cb P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.526826 2526 tablet_replica.cc:331] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.527015 1158 ts_tablet_manager.cc:1918] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.527027 2526 raft_consensus.cc:2241] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.527149 2526 raft_consensus.cc:2270] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.527158 1158 log.cc:1199] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/50d23f589de84dd5b150a17cee00c411
I20250629 01:52:01.527415 1158 ts_tablet_manager.cc:1939] T 50d23f589de84dd5b150a17cee00c411 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.527681 2526 ts_tablet_manager.cc:1905] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.529127 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet cb7314fbdfc74b1c8d9d1be9c05244cb (table default.Table_f66f05a84a344ef2ab6fd214c4dc4cc0 [id=0673c093fb474e9bb969c584dd00e4f9]) successfully deleted
I20250629 01:52:01.529714 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_eca0690b56fd472e8b2baf252d3721ca [id=e735a7d5da48458da86326bf01640f70] alter complete (version 3)
I20250629 01:52:01.530457 2487 tablet.cc:1722] T 1bc3d470df0d4814a655bcff68c956e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:01.530834 1158 tablet_replica.cc:331] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.530979 1158 raft_consensus.cc:2241] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.531076 1158 raft_consensus.cc:2270] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.531445 1158 ts_tablet_manager.cc:1905] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:01.533062 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57eb4e64e9e547778f1a6acf37e6f251 consensus state has no leader
I20250629 01:52:01.533456 1155 tablet_bootstrap.cc:492] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.533545 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ee7fb74854db4034a9d44f6d025e6756
I20250629 01:52:01.533562 1155 ts_tablet_manager.cc:1397] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.005s
I20250629 01:52:01.534085 1155 raft_consensus.cc:357] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.534308 1155 raft_consensus.cc:383] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.534364 1155 raft_consensus.cc:738] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.534525 1155 consensus_queue.cc:260] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.534920 1155 ts_tablet_manager.cc:1428] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:01.535130 1155 tablet_bootstrap.cc:492] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.535840 2580 raft_consensus.cc:491] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.535935 2580 raft_consensus.cc:513] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.536286 1155 tablet_bootstrap.cc:654] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.536386 2580 leader_election.cc:290] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.536643 896 tablet_service.cc:1515] Processing DeleteTablet for tablet ee7fb74854db4034a9d44f6d025e6756 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.537388 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 50d23f589de84dd5b150a17cee00c411 (table default.Table_56ce186b15474e3ebf35ac8f454778bf [id=5a9ebb3e7cd54996a1d7e312bcc3d8fb]) successfully deleted
I20250629 01:52:01.537508 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1348) took 1027 ms. Trace:
I20250629 01:52:01.537609 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet ee7fb74854db4034a9d44f6d025e6756 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.537712 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 50d23f589de84dd5b150a17cee00c411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.537900 740 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.538018 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.538067 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.538146 742 tablet_service.cc:1515] Processing DeleteTablet for tablet ee7fb74854db4034a9d44f6d025e6756 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.537580 1156 rpcz_store.cc:276] 0629 01:52:00.510361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.510606 (+ 245us) service_pool.cc:225] Handling call
0629 01:52:01.283939 (+773333us) ts_tablet_manager.cc:1145] Deleting tablet 1b0782c7e652419096710563930f2072
0629 01:52:01.283947 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.455073 (+171126us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.457486 (+ 2413us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.537465 (+ 79979us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":760043}
I20250629 01:52:01.540654 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b423766fabef4323aaa0a999696efb58" } modify_external_catalogs: true
I20250629 01:52:01.540824 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b423766fabef4323aaa0a999696efb58" } modify_external_catalogs: true
I20250629 01:52:01.541216 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 1b0782c7e652419096710563930f2072 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
W20250629 01:52:01.541383 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24dd74d425a74e79894f309117372ebf consensus state has no leader
I20250629 01:52:01.541672 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cf77f01c09db4169a6965146a7a98f10" } modify_external_catalogs: true
I20250629 01:52:01.541779 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cf77f01c09db4169a6965146a7a98f10" } modify_external_catalogs: true
I20250629 01:52:01.542284 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 0c422ceed7594a18b32e5f70350e5b7b from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:01.542330 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d57756db239b4178a430514058de8274" } new_table_name: "default.Table_efe191859d3a466d910153a9039f26a8" modify_external_catalogs: true
I20250629 01:52:01.542519 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 898b49812ece4b57a1a1015d9f26be2f with delete_type TABLET_DATA_DELETED (replaced by tablet 573b329d020549a08c94891916164aeb) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.542673 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e0727102d54294a7c13a4aeef311e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.542764 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet dc841eb171d04b118b1e705ceed4c861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.542799 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 7908afa2abaa4521b44f954e0790f986 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.542881 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_49b84436278348afa56cf408cab52a43" } new_table_name: "default.Table_e9d6196808134c678df6ee3b7d7349f3" modify_external_catalogs: true
I20250629 01:52:01.542938 1156 tablet_replica.cc:331] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:01.543308 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.543809 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d1dc5307aa7c47738a408a2ebd00a685" } new_table_name: "default.Table_78a3a28950904cda8cd52ea3e249896d" modify_external_catalogs: true
I20250629 01:52:01.544370 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.544750 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_396c6df1e4db40aeae8e6980af7acc13" } modify_external_catalogs: true
I20250629 01:52:01.544873 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_396c6df1e4db40aeae8e6980af7acc13" } modify_external_catalogs: true
I20250629 01:52:01.546234 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9790ad1c1a4447949cfe0e98c46458df" } modify_external_catalogs: true
I20250629 01:52:01.546350 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9790ad1c1a4447949cfe0e98c46458df" } modify_external_catalogs: true
I20250629 01:52:01.547428 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c64d8f4e6caa4ef4a4feb26802b0fbd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.548415 1158 ts_tablet_manager.cc:1918] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.548493 1158 log.cc:1199] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/3bdc6a26b3594d6fb33a27cac5b29f4f
I20250629 01:52:01.548730 1158 ts_tablet_manager.cc:1939] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.549347 2487 tablet.cc:1722] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:01.551182 1155 tablet_bootstrap.cc:492] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.551472 1155 ts_tablet_manager.cc:1397] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250629 01:52:01.549479 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 3bdc6a26b3594d6fb33a27cac5b29f4f (table default.Table_99703cee14e34f2ebe6028b072a5b072 [id=2ba8a990fc3a474aac09caa7380ccb80]) successfully deleted
I20250629 01:52:01.552362 1155 raft_consensus.cc:357] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.552558 1155 raft_consensus.cc:383] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.552659 1155 raft_consensus.cc:738] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.552857 1155 consensus_queue.cc:260] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.553372 1155 ts_tablet_manager.cc:1428] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.553640 1155 tablet_bootstrap.cc:492] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.554642 449 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:52:01.554778 1155 tablet_bootstrap.cc:654] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.555383 2670 raft_consensus.cc:2747] T 9c799a04c0f74492b9fb674f9b0da240 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.555680 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.557161 1156 raft_consensus.cc:2241] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.557333 1156 raft_consensus.cc:2270] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.557600 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc841eb171d04b118b1e705ceed4c861" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.557796 1156 ts_tablet_manager.cc:1905] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:01.558108 988 leader_election.cc:343] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.558241 2580 raft_consensus.cc:491] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.558331 2580 raft_consensus.cc:513] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.558729 2580 leader_election.cc:290] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.559329 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aa8dfd712c043ad9f8ff56b064bb8fd" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.559674 988 leader_election.cc:343] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.560644 1154 tablet_bootstrap.cc:492] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.560760 1154 ts_tablet_manager.cc:1397] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.086s user 0.000s sys 0.003s
I20250629 01:52:01.561300 1154 raft_consensus.cc:357] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.561408 1154 raft_consensus.cc:383] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.561443 1154 raft_consensus.cc:738] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.561589 1154 consensus_queue.cc:260] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.561764 2487 tablet.cc:1722] T 347fa6a12a744ee88058643b7143e39d P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:52:01.561955 1154 ts_tablet_manager.cc:1428] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:01.562163 1154 tablet_bootstrap.cc:492] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.562963 1156 ts_tablet_manager.cc:1918] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.563043 1156 log.cc:1199] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e4d6f1eb43b24fda94aa0afcf4501195
I20250629 01:52:01.563259 1156 ts_tablet_manager.cc:1939] T e4d6f1eb43b24fda94aa0afcf4501195 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:01.563388 1154 tablet_bootstrap.cc:654] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.565589 1158 tablet_replica.cc:331] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.565730 1158 raft_consensus.cc:2241] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.565896 1158 pending_rounds.cc:70] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:01.565928 1158 pending_rounds.cc:77] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759227204870144 op_type: NO_OP noop_request { }
I20250629 01:52:01.565991 1158 raft_consensus.cc:2887] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.566051 1158 raft_consensus.cc:2270] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:01.566045 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0625c1d047504984b8d271004d40f15c consensus state has no leader
I20250629 01:52:01.566613 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1349) took 1056 ms. Trace:
I20250629 01:52:01.566679 1156 rpcz_store.cc:276] 0629 01:52:00.510461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.510721 (+ 260us) service_pool.cc:225] Handling call
0629 01:52:01.542900 (+1032179us) ts_tablet_manager.cc:1145] Deleting tablet e4d6f1eb43b24fda94aa0afcf4501195
0629 01:52:01.542907 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.560530 (+ 17623us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.562902 (+ 2372us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.566598 (+ 3696us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14064,"tablet-delete.queue_time_us":1018974}
I20250629 01:52:01.567106 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e4d6f1eb43b24fda94aa0afcf4501195 (table default.Table_1e578e5fe38c40d38ba9b7759798a267 [id=64e6f3a1044a485dac278b2deccc979d]) successfully deleted
W20250629 01:52:01.567164 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12efda19b5ee445f968a5671522295f5 consensus state has no leader
W20250629 01:52:01.568832 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fac9c587cca42ab8f027b8dc01a79f3 consensus state has no leader
I20250629 01:52:01.569128 1158 ts_tablet_manager.cc:1905] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.570333 2527 raft_consensus.cc:491] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.570479 1156 tablet_replica.cc:331] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:01.570434 2527 raft_consensus.cc:513] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.570621 1156 raft_consensus.cc:2241] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.570794 1156 pending_rounds.cc:70] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:01.570822 1156 pending_rounds.cc:77] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759226478333952 op_type: NO_OP noop_request { }
I20250629 01:52:01.570879 1156 raft_consensus.cc:2887] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.570879 2527 leader_election.cc:290] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.570930 1156 raft_consensus.cc:2270] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.571699 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:01.572907 988 leader_election.cc:343] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.576368 1156 ts_tablet_manager.cc:1905] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.581162 1155 tablet_bootstrap.cc:492] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.581301 1155 ts_tablet_manager.cc:1397] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250629 01:52:01.581810 1155 raft_consensus.cc:357] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.581981 1155 raft_consensus.cc:383] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.582052 1155 raft_consensus.cc:738] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.582221 1155 consensus_queue.cc:260] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.582618 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.582731 1155 ts_tablet_manager.cc:1428] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:01.582971 1155 tablet_bootstrap.cc:492] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.583999 1155 tablet_bootstrap.cc:654] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.584556 1159 tablet_bootstrap.cc:492] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.584709 1159 ts_tablet_manager.cc:1397] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.169s user 0.005s sys 0.010s
W20250629 01:52:01.584980 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c64d8f4e6caa4ef4a4feb26802b0fbd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.585330 1159 raft_consensus.cc:357] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.585459 1159 raft_consensus.cc:383] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.585500 1159 raft_consensus.cc:738] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.585682 1159 consensus_queue.cc:260] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.585852 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 347fa6a12a744ee88058643b7143e39d (table default.Table_923010b6578e479089f2b7da4e770d9a [id=0fb1390e9626454eba8e8d5c634e3943]). Expected version 5 got 0
I20250629 01:52:01.586265 455 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:52:01.587661 1159 ts_tablet_manager.cc:1428] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:52:01.587875 1159 tablet_bootstrap.cc:492] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.588152 2526 ts_tablet_manager.cc:1918] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.588234 2526 log.cc:1199] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/20586ba2af9e4cb7adec5ca6c3b76e42
I20250629 01:52:01.588774 2526 ts_tablet_manager.cc:1939] T 20586ba2af9e4cb7adec5ca6c3b76e42 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.589114 2526 tablet_replica.cc:331] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.589222 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 20586ba2af9e4cb7adec5ca6c3b76e42 (table default.Table_380dba3955bd458caa9a65e48c00f407 [id=77b6e244496149c5a3e46fdfb5bbe791]) successfully deleted
I20250629 01:52:01.589257 2526 raft_consensus.cc:2241] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.589390 2526 raft_consensus.cc:2270] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.589442 896 tablet_service.cc:1515] Processing DeleteTablet for tablet fbcae02d893f4795a12d26af57b18b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.589504 1158 ts_tablet_manager.cc:1918] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.589586 1158 log.cc:1199] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/303f77a729c648f9b1146d11056b242d
I20250629 01:52:01.589604 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 6092e19acf974f2dad5a903da72944ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.589717 2580 raft_consensus.cc:491] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.589738 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 40592f11f430469f9c8e82eba25b38ed with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.589811 2526 ts_tablet_manager.cc:1905] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.589854 1158 ts_tablet_manager.cc:1939] T 303f77a729c648f9b1146d11056b242d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.589855 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 68cedbd093b4420d8f7b29219ce8f77b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:44 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.589811 2580 raft_consensus.cc:513] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.590278 2580 leader_election.cc:290] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.591251 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a9d2aa33a5c4d9ebecaa83c32b39dcb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.593096 877 tablet_service.cc:1515] Processing DeleteTablet for tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.593252 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 74b3badd684d45fa88dc8cbcfc61001a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.593349 1156 ts_tablet_manager.cc:1918] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.593426 1156 log.cc:1199] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/f94d8fc1b6254ba69acadcaff3613d97
I20250629 01:52:01.593713 1156 ts_tablet_manager.cc:1939] T f94d8fc1b6254ba69acadcaff3613d97 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:01.594777 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 303f77a729c648f9b1146d11056b242d (table default.Table_fcb861de87ad468584f2f0c88b9db3b6 [id=7c4d6f0956f142d195bd44428072be8f]) successfully deleted
I20250629 01:52:01.595377 1158 tablet_replica.cc:331] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.595693 1158 raft_consensus.cc:2241] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.596007 1158 raft_consensus.cc:2270] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.596592 1158 ts_tablet_manager.cc:1905] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.597268 1159 tablet_bootstrap.cc:654] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:01.596765 988 leader_election.cc:343] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.599267 2651 raft_consensus.cc:491] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.599373 2651 raft_consensus.cc:513] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.599699 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1350) took 1089 ms. Trace:
I20250629 01:52:01.599691 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76118a6e684448b09f244d44c909ccaa" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.599817 2651 leader_election.cc:290] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.600162 896 tablet_service.cc:1515] Processing DeleteTablet for tablet e3b94f8250a143b096703f2b0386e3cd with delete_type TABLET_DATA_DELETED (replaced by tablet c5771fdd618e45118df2b6dab7f668d9) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.600674 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.600855 1052 raft_consensus.cc:2466] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:01.601207 710 leader_election.cc:304] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.601486 2651 raft_consensus.cc:2802] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.601547 2651 raft_consensus.cc:491] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.601598 2651 raft_consensus.cc:3058] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.603818 2651 raft_consensus.cc:513] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.599764 1156 rpcz_store.cc:276] 0629 01:52:00.510563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.510810 (+ 247us) service_pool.cc:225] Handling call
0629 01:52:01.570442 (+1059632us) ts_tablet_manager.cc:1145] Deleting tablet f94d8fc1b6254ba69acadcaff3613d97
0629 01:52:01.570450 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.578767 (+ 8317us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.593283 (+ 14516us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:01.599685 (+ 6402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046459}
I20250629 01:52:01.604838 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet f94d8fc1b6254ba69acadcaff3613d97 (table default.Table_3fc2c8b5061f4a6e8f46cd439cf5c776 [id=92d55fc2cf6343d7aac796eb0ddb93f8]) successfully deleted
I20250629 01:52:01.605195 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1351) took 1012 ms. Trace:
I20250629 01:52:01.605511 1156 rpcz_store.cc:276] 0629 01:52:00.592787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:00.594350 (+ 1563us) service_pool.cc:225] Handling call
0629 01:52:01.605100 (+1010750us) ts_tablet_manager.cc:1145] Deleting tablet b5bcfb2ac1674beb8e9d50d2818cec0d
0629 01:52:01.605107 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:01.605181 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1010565}
W20250629 01:52:01.606012 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet b5bcfb2ac1674beb8e9d50d2818cec0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5bcfb2ac1674beb8e9d50d2818cec0d already in progress: creating tablet
I20250629 01:52:01.606216 2651 leader_election.cc:290] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:01.606670 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e3874404c3444d47a71f1dce7fb86ff8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3874404c3444d47a71f1dce7fb86ff8 already in progress: creating tablet
I20250629 01:52:01.606659 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76118a6e684448b09f244d44c909ccaa" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.607034 2651 raft_consensus.cc:491] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.607139 2651 raft_consensus.cc:513] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:52:01.607481 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet bae91d28f27f4e14a23c65e41091b671 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bae91d28f27f4e14a23c65e41091b671 already in progress: creating tablet
I20250629 01:52:01.607864 2651 leader_election.cc:290] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:01.607985 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 8ae366e6ccf34a9faf2caa2bded63546 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ae366e6ccf34a9faf2caa2bded63546 already in progress: creating tablet
I20250629 01:52:01.608222 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecbfc4e45dc9435bbd7a55ba7b7e4155" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.608412 1051 raft_consensus.cc:2466] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:52:01.608654 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet c2591c65d53449c8801778606f1f05f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2591c65d53449c8801778606f1f05f1 already in progress: creating tablet
I20250629 01:52:01.608685 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.608696 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.608790 1052 raft_consensus.cc:3058] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.608870 710 leader_election.cc:304] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.609073 2534 raft_consensus.cc:2802] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.609127 2534 raft_consensus.cc:491] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.609172 2534 raft_consensus.cc:3058] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.610987 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e0727102d54294a7c13a4aeef311e1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.611119 773 raft_consensus.cc:2466] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:01.611220 2534 raft_consensus.cc:513] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.611346 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecbfc4e45dc9435bbd7a55ba7b7e4155" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.611446 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76118a6e684448b09f244d44c909ccaa" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.611517 774 raft_consensus.cc:2391] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:52:01.611670 2534 leader_election.cc:290] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.611690 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aa8dfd712c043ad9f8ff56b064bb8fd" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.611995 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc841eb171d04b118b1e705ceed4c861" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.612546 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecbfc4e45dc9435bbd7a55ba7b7e4155" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.612666 1048 raft_consensus.cc:3058] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.613109 1156 tablet_replica.cc:331] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:01.613257 1156 raft_consensus.cc:2241] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.613469 1156 raft_consensus.cc:2270] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.613862 1156 ts_tablet_manager.cc:1905] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.614837 1048 raft_consensus.cc:2466] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:01.615513 710 leader_election.cc:304] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:52:01.616065 710 consensus_peers.cc:489] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a014907d09c94745859436f1eb73b280. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:01.616190 1052 raft_consensus.cc:2466] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:01.616250 2651 raft_consensus.cc:2802] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.616345 2651 raft_consensus.cc:695] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.616627 2651 consensus_queue.cc:237] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.617125 710 leader_election.cc:304] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.617465 2649 raft_consensus.cc:2802] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.617559 2649 raft_consensus.cc:695] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.617750 2649 consensus_queue.cc:237] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.618147 1158 ts_tablet_manager.cc:1918] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.618237 1158 log.cc:1199] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/5ab0aa4e74d345b2a214b90b5018c82b
I20250629 01:52:01.618491 1158 ts_tablet_manager.cc:1939] T 5ab0aa4e74d345b2a214b90b5018c82b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.619302 1158 tablet_replica.cc:331] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.619403 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 5ab0aa4e74d345b2a214b90b5018c82b (table default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]) successfully deleted
I20250629 01:52:01.619467 1158 raft_consensus.cc:2241] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.619608 1158 raft_consensus.cc:2270] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.620081 1158 ts_tablet_manager.cc:1905] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.632869 1155 tablet_bootstrap.cc:492] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.633003 1155 ts_tablet_manager.cc:1397] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.002s
I20250629 01:52:01.633508 1155 raft_consensus.cc:357] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.633708 1155 raft_consensus.cc:383] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.633782 1155 raft_consensus.cc:738] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.633981 1155 consensus_queue.cc:260] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.634639 1155 ts_tablet_manager.cc:1428] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:01.634881 1155 tablet_bootstrap.cc:492] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.636029 1155 tablet_bootstrap.cc:654] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.636279 1154 tablet_bootstrap.cc:492] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.636476 1154 ts_tablet_manager.cc:1397] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.006s
I20250629 01:52:01.637300 1154 raft_consensus.cc:357] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.637537 1154 raft_consensus.cc:383] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.637646 1154 raft_consensus.cc:738] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.637959 2046 generic_service.cc:196] Checking for leaks (request via RPC)
I20250629 01:52:01.640440 1156 ts_tablet_manager.cc:1918] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.640571 1156 log.cc:1199] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/adbb648c49e24b8085eacbe6e0a86991
W20250629 01:52:01.641045 712 consensus_peers.cc:489] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:01.641258 1156 ts_tablet_manager.cc:1939] T adbb648c49e24b8085eacbe6e0a86991 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
W20250629 01:52:01.643994 712 leader_election.cc:343] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:01.644279 712 leader_election.cc:343] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:01.644833 712 leader_election.cc:343] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.644922 712 leader_election.cc:304] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:01.645097 2649 raft_consensus.cc:2747] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.645385 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.646147 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.646214 1154 consensus_queue.cc:260] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.646452 986 leader_election.cc:304] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.646806 986 leader_election.cc:304] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.647133 2580 raft_consensus.cc:2802] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.647207 2580 raft_consensus.cc:491] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.647269 2580 raft_consensus.cc:3058] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.647127 1154 ts_tablet_manager.cc:1428] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250629 01:52:01.647610 1154 tablet_bootstrap.cc:492] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.649618 2580 raft_consensus.cc:513] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.649892 1154 tablet_bootstrap.cc:654] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.650276 2527 raft_consensus.cc:2747] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.650681 2580 leader_election.cc:290] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.651314 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecbfc4e45dc9435bbd7a55ba7b7e4155" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.651701 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ecbfc4e45dc9435bbd7a55ba7b7e4155" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.655246 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e0727102d54294a7c13a4aeef311e1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.655309 1158 ts_tablet_manager.cc:1918] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.655387 1158 log.cc:1199] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/e5d27d658892496a9b19b57949a8399a
W20250629 01:52:01.655604 988 leader_election.cc:343] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.657492 1159 tablet_bootstrap.cc:492] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.657563 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.657696 1159 ts_tablet_manager.cc:1397] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20250629 01:52:01.654541 1155 tablet_bootstrap.cc:492] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.658110 1155 ts_tablet_manager.cc:1397] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20250629 01:52:01.658522 1159 raft_consensus.cc:357] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.658614 1155 raft_consensus.cc:357] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.658740 1155 raft_consensus.cc:383] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.658718 1159 raft_consensus.cc:383] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.658794 1155 raft_consensus.cc:738] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.658880 1159 raft_consensus.cc:738] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.658942 1155 consensus_queue.cc:260] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.659915 2526 ts_tablet_manager.cc:1918] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.660032 2526 log.cc:1199] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9cfff25bea764e62954e49611b71703c
I20250629 01:52:01.660144 1159 consensus_queue.cc:260] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.660449 1155 ts_tablet_manager.cc:1428] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:01.660884 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a9d2aa33a5c4d9ebecaa83c32b39dcb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:01.661336 777 raft_consensus.cc:2466] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:01.661162 1155 tablet_bootstrap.cc:492] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:01.664165 712 leader_election.cc:343] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.664254 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2ed6d41c999f40d994a3609923eeffd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.664382 712 leader_election.cc:343] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:01.664896 712 leader_election.cc:343] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:01.665865 712 leader_election.cc:343] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.666203 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 258994dd24804f4aaa5e299436fac9b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.666623 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b80c833964504cee9ae5d15210febf5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.667884 1155 tablet_bootstrap.cc:654] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.669142 2526 ts_tablet_manager.cc:1939] T 9cfff25bea764e62954e49611b71703c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.669721 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9cfff25bea764e62954e49611b71703c (table default.Table_bf7b726d4c38474994e10c22d1aa6076 [id=875222c495154458ae2052f4ca5d220f]) successfully deleted
I20250629 01:52:01.669966 2526 tablet_replica.cc:331] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:52:01.669957 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet ee7fb74854db4034a9d44f6d025e6756 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7fb74854db4034a9d44f6d025e6756 already in progress: creating tablet
I20250629 01:52:01.670111 2526 raft_consensus.cc:2241] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.670217 2526 raft_consensus.cc:2270] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.670636 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8147f0891d8a40e185acaad58f89fc75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.671581 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b80c833964504cee9ae5d15210febf5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:01.671590 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ed6d41c999f40d994a3609923eeffd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:01.673389 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8147f0891d8a40e185acaad58f89fc75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.674286 2526 ts_tablet_manager.cc:1905] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.675237 1159 ts_tablet_manager.cc:1428] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250629 01:52:01.655594 1158 ts_tablet_manager.cc:1939] T e5d27d658892496a9b19b57949a8399a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.678825 1158 tablet_replica.cc:331] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.678974 1158 raft_consensus.cc:2241] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.679091 1158 raft_consensus.cc:2270] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.679483 1158 ts_tablet_manager.cc:1905] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.680238 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet e5d27d658892496a9b19b57949a8399a (table default.Table_768dd2c93bfa4a828037ed23b11dc5ae [id=e8832475ef554a2ebbf61f3481bf197c]) successfully deleted
I20250629 01:52:01.681072 1155 tablet_bootstrap.cc:492] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.681174 1155 ts_tablet_manager.cc:1397] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20250629 01:52:01.681686 1155 raft_consensus.cc:357] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.681808 1155 raft_consensus.cc:383] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.681851 1155 raft_consensus.cc:738] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.682000 1155 consensus_queue.cc:260] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.683352 1155 ts_tablet_manager.cc:1428] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:01.683557 1155 tablet_bootstrap.cc:492] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.683777 1159 tablet_bootstrap.cc:492] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.684841 1156 tablet_replica.cc:331] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:01.684866 1155 tablet_bootstrap.cc:654] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.684984 1156 raft_consensus.cc:2241] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.685115 1159 tablet_bootstrap.cc:654] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.685222 1156 pending_rounds.cc:70] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:01.685266 1156 pending_rounds.cc:77] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759226201767936 op_type: NO_OP noop_request { }
I20250629 01:52:01.685354 1156 raft_consensus.cc:2887] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.685415 1156 raft_consensus.cc:2270] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.685825 1156 ts_tablet_manager.cc:1905] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.686389 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet adbb648c49e24b8085eacbe6e0a86991 (table default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]) successfully deleted
I20250629 01:52:01.690021 1154 tablet_bootstrap.cc:492] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.693389 1156 ts_tablet_manager.cc:1918] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.693524 1156 log.cc:1199] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a014907d09c94745859436f1eb73b280
I20250629 01:52:01.693825 1156 ts_tablet_manager.cc:1939] T a014907d09c94745859436f1eb73b280 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:01.694691 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a014907d09c94745859436f1eb73b280 (table default.Table_98056b52f364497899c1998d23f58ce7 [id=d63189e5f0564d20a7f992ec881309fd]) successfully deleted
W20250629 01:52:01.695432 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 7edfdf771ecd4ae8b87791846536271f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7edfdf771ecd4ae8b87791846536271f already in progress: creating tablet
W20250629 01:52:01.690439 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 43649.6us
I20250629 01:52:01.695923 986 leader_election.cc:304] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.696339 2526 ts_tablet_manager.cc:1918] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.696427 2526 log.cc:1199] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/fbcae02d893f4795a12d26af57b18b85
W20250629 01:52:01.696447 986 leader_election.cc:343] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.696525 986 leader_election.cc:304] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.696671 2526 ts_tablet_manager.cc:1939] T fbcae02d893f4795a12d26af57b18b85 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:52:01.696774 986 leader_election.cc:343] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.696833 986 leader_election.cc:304] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.697206 2580 raft_consensus.cc:2747] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.697330 2580 raft_consensus.cc:2747] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.697350 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet fbcae02d893f4795a12d26af57b18b85 (table default.Table_9d5bbbb0daad4acca9c32a66af5f2ae2 [id=a128a46807814758b9b79071ca6db923]) successfully deleted
I20250629 01:52:01.697425 2580 raft_consensus.cc:2747] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.697613 2526 tablet_replica.cc:331] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.697759 2526 raft_consensus.cc:2241] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.697865 2526 raft_consensus.cc:2270] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:01.698251 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 6b4d4d6dacb84f80903522d20de4e795 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4d4d6dacb84f80903522d20de4e795 already in progress: creating tablet
I20250629 01:52:01.698897 2526 ts_tablet_manager.cc:1905] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.702968 986 leader_election.cc:304] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:01.703326 986 leader_election.cc:304] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:01.704219 1154 ts_tablet_manager.cc:1397] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.001s
I20250629 01:52:01.704520 1155 tablet_bootstrap.cc:492] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.704622 1155 ts_tablet_manager.cc:1397] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20250629 01:52:01.705140 1155 raft_consensus.cc:357] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.705272 1155 raft_consensus.cc:383] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.705325 1155 raft_consensus.cc:738] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.705492 1155 consensus_queue.cc:260] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.705996 1155 ts_tablet_manager.cc:1428] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:01.708787 2762 raft_consensus.cc:2747] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:01.708863 2527 raft_consensus.cc:2802] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.708915 2527 raft_consensus.cc:491] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.708966 2527 raft_consensus.cc:3058] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.709692 912 raft_consensus.cc:3058] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.710587 2765 tablet.cc:1722] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:01.713563 1155 tablet_bootstrap.cc:492] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.714025 2527 raft_consensus.cc:513] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.714807 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a9d2aa33a5c4d9ebecaa83c32b39dcb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.715119 2487 tablet.cc:1722] T e1e22a0c02054c7db514eefe20ee3311 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:52:01.715221 988 leader_election.cc:343] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.715955 2527 leader_election.cc:290] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.718242 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 3a81e6f9711c4d398ae09f33a75566ae from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:01.720400 1159 tablet_bootstrap.cc:492] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.720851 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07e0727102d54294a7c13a4aeef311e1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:01.720980 777 raft_consensus.cc:3058] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.723212 777 raft_consensus.cc:2466] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:01.726042 1159 ts_tablet_manager.cc:1397] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250629 01:52:01.726809 1159 raft_consensus.cc:357] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.724488 2526 ts_tablet_manager.cc:1918] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.729560 2526 log.cc:1199] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/6092e19acf974f2dad5a903da72944ba
I20250629 01:52:01.730007 2526 ts_tablet_manager.cc:1939] T 6092e19acf974f2dad5a903da72944ba P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.730829 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 6092e19acf974f2dad5a903da72944ba (table default.Table_913feb22c480457e96599a6814b30c60 [id=abbf099fb0aa4b118f76e5c48700df39]) successfully deleted
I20250629 01:52:01.731047 2526 tablet_replica.cc:331] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.731233 2526 raft_consensus.cc:2241] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.731379 2526 raft_consensus.cc:2270] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.731586 1159 raft_consensus.cc:383] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.731734 1159 raft_consensus.cc:738] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.732147 1154 raft_consensus.cc:357] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.732035 1159 consensus_queue.cc:260] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.734324 1154 raft_consensus.cc:383] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.734385 1154 raft_consensus.cc:738] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.734525 1154 consensus_queue.cc:260] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.734948 1159 ts_tablet_manager.cc:1428] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250629 01:52:01.734894 1154 ts_tablet_manager.cc:1428] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.031s user 0.000s sys 0.000s
I20250629 01:52:01.735792 2526 ts_tablet_manager.cc:1905] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.736040 1154 tablet_bootstrap.cc:492] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.737331 1154 tablet_bootstrap.cc:654] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.737655 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_76b76a14142749efb0305e749805438f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.737756 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_50ba0d7936d04f23b5eced9bd61474b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.738129 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50ba0d7936d04f23b5eced9bd61474b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.739073 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9f57ddf2fcf445128766cfb7fcc80f3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.739444 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f57ddf2fcf445128766cfb7fcc80f3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.732335 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.732511 2763 tablet.cc:1722] T e1e22a0c02054c7db514eefe20ee3311 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:52:01.738129 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76b76a14142749efb0305e749805438f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.745308 1159 tablet_bootstrap.cc:492] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.750039 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_620b12abd6f9476e8bf5127f618d9295" } modify_external_catalogs: true
I20250629 01:52:01.750178 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_620b12abd6f9476e8bf5127f618d9295" } modify_external_catalogs: true
I20250629 01:52:01.750758 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f802e382de6348e9b94cf6982926fa33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.751124 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f802e382de6348e9b94cf6982926fa33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.753401 451 catalog_manager.cc:5582] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:01.753715 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 76118a6e684448b09f244d44c909ccaa (table default.Table_5794ea9989b54ab9bd7ecca30428c971 [id=dcc07a46945a4add8e667bfea51dfed7]). Expected version 5 got 0
I20250629 01:52:01.753933 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:01.754053 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:01.754182 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:52:01.754312 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:01.758262 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 6a3f9c4dfce34596b5145cfeb887f727 (DEFAULT_TABLE table=default.Table_f3cd4aaddc584fb39053963cf9005c3d [id=5b2efc95946e4937b22e95b2fa4685e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.758399 1158 ts_tablet_manager.cc:1918] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.758467 1158 log.cc:1199] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ecbfc4e45dc9435bbd7a55ba7b7e4155
I20250629 01:52:01.758574 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3f9c4dfce34596b5145cfeb887f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.758731 1158 ts_tablet_manager.cc:1939] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.759459 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 (table default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]) successfully deleted
I20250629 01:52:01.761592 2526 ts_tablet_manager.cc:1918] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.761668 2526 log.cc:1199] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/40592f11f430469f9c8e82eba25b38ed
I20250629 01:52:01.762826 1159 tablet_bootstrap.cc:654] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.762879 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3 (DEFAULT_TABLE table=default.Table_c24aa328eefc49d6979d74d3945cc219 [id=357d8ab96405410fbd77680046a87e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.763156 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.763772 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 8965a607d2a141ec94d9fdbf605f4bd7 (DEFAULT_TABLE table=default.Table_3e0a0e61a2314bdba910cdd875e456bf [id=e77821f698bd41d5ad96c03c31c58f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.764036 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8965a607d2a141ec94d9fdbf605f4bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.767912 742 tablet_service.cc:1468] Processing CreateTablet for tablet 6a3f9c4dfce34596b5145cfeb887f727 (DEFAULT_TABLE table=default.Table_f3cd4aaddc584fb39053963cf9005c3d [id=5b2efc95946e4937b22e95b2fa4685e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.767913 747 tablet_service.cc:1468] Processing CreateTablet for tablet 8965a607d2a141ec94d9fdbf605f4bd7 (DEFAULT_TABLE table=default.Table_3e0a0e61a2314bdba910cdd875e456bf [id=e77821f698bd41d5ad96c03c31c58f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.769307 1154 tablet_bootstrap.cc:492] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.769400 2526 ts_tablet_manager.cc:1939] T 40592f11f430469f9c8e82eba25b38ed P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.769454 1154 ts_tablet_manager.cc:1397] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.004s
I20250629 01:52:01.769784 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4ee1e7985f4b4422ba607962bf1fdc42
I20250629 01:52:01.770103 1154 raft_consensus.cc:357] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.770287 1154 raft_consensus.cc:383] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.770365 1154 raft_consensus.cc:738] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.770328 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af4707cebd704b4ba5a48ab600ab26f0" } new_table_name: "default.Table_35b86775da724816860723eec557b02e" modify_external_catalogs: true
I20250629 01:52:01.770594 1154 consensus_queue.cc:260] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.771025 1154 ts_tablet_manager.cc:1428] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:01.771260 1154 tablet_bootstrap.cc:492] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.772298 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ee1e7985f4b4422ba607962bf1fdc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.772755 2526 tablet_replica.cc:331] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.772774 1154 tablet_bootstrap.cc:654] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.772943 1158 tablet_replica.cc:331] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.773644 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3f9c4dfce34596b5145cfeb887f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.774114 1158 raft_consensus.cc:2241] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:01.775534 1158 pending_rounds.cc:70] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Trying to abort 1 pending ops.
I20250629 01:52:01.775588 1158 pending_rounds.cc:77] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759229031493632 op_type: NO_OP noop_request { }
I20250629 01:52:01.775650 1158 raft_consensus.cc:2887] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:01.775946 2526 raft_consensus.cc:2241] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.776338 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8965a607d2a141ec94d9fdbf605f4bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.778481 1159 tablet_bootstrap.cc:492] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.778582 1159 ts_tablet_manager.cc:1397] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20250629 01:52:01.779165 1159 raft_consensus.cc:357] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.779282 1159 raft_consensus.cc:383] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.779315 1159 raft_consensus.cc:738] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.776054 2526 raft_consensus.cc:2270] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.780277 1158 raft_consensus.cc:2270] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.781157 1158 ts_tablet_manager.cc:1905] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.784494 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e45d1c8068e346039524312a058e9de7
I20250629 01:52:01.784893 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 8688bcb3523f4a7fa9366f9b648274e8 (DEFAULT_TABLE table=default.Table_b614c356b8dd4e5d854d84b62f1ff989 [id=e2afb5601a81486891b231f84ea74239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.785049 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_096d8cb0cba44f5db475e2badfbea1b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.785249 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8688bcb3523f4a7fa9366f9b648274e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:01.785415 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_096d8cb0cba44f5db475e2badfbea1b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.785908 2526 ts_tablet_manager.cc:1905] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.787151 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ee1e7985f4b4422ba607962bf1fdc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.779445 1159 consensus_queue.cc:260] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.787525 740 tablet_service.cc:1468] Processing CreateTablet for tablet c2f0c1c26c49457db605222093b367cf (DEFAULT_TABLE table=default.Table_8ccb29231d6746ab93412d34b939117f [id=9bad87286c4a4850b11d8f49a8a46d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.787817 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f0c1c26c49457db605222093b367cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.787823 1159 ts_tablet_manager.cc:1428] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250629 01:52:01.788038 1159 tablet_bootstrap.cc:492] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.781937 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 0c9c652531df452c963691ff3ac5fb7e (DEFAULT_TABLE table=default.Table_704f6c68c7484e5582a42cef6e942588 [id=c746a9e9c88844909d8409b125863b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.788021 751 tablet_service.cc:1468] Processing CreateTablet for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3 (DEFAULT_TABLE table=default.Table_c24aa328eefc49d6979d74d3945cc219 [id=357d8ab96405410fbd77680046a87e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.788288 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9c652531df452c963691ff3ac5fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.788426 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.781960 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 81e67d94ba304f9eb030b854fc17d0d7 (DEFAULT_TABLE table=default.Table_e033aedd05ca4aef91836f468c2adfb0 [id=f74d9b675ea940259a17ba01881d3292]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.792302 750 tablet_service.cc:1468] Processing CreateTablet for tablet 1966a23b1fe14e6a94bd24ee4e8447a0 (DEFAULT_TABLE table=default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6 [id=8f6f7d0a2f4f43b8a3163e41db7e7351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.792481 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e67d94ba304f9eb030b854fc17d0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.792598 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1966a23b1fe14e6a94bd24ee4e8447a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.794886 1158 ts_tablet_manager.cc:1918] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:01.794960 1158 log.cc:1199] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/8295e8178eff4d5eb95ac885092731e0
I20250629 01:52:01.795233 1158 ts_tablet_manager.cc:1939] T 8295e8178eff4d5eb95ac885092731e0 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:01.795989 1158 tablet_replica.cc:331] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:01.782071 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 24f980dafd8243738236cb8a0491770c (DEFAULT_TABLE table=default.Table_3e8277c1c24c4c58aba9b1f245d490c6 [id=3e9b6b745e5c4911984bb347687410de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.796334 753 tablet_service.cc:1468] Processing CreateTablet for tablet d331a13d13f34456be057dd9cc9fc23b (DEFAULT_TABLE table=default.Table_dc29904c04dd44f9b8f1dc8bdf527229 [id=0c63248009934926aa42aace97554161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.796411 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f980dafd8243738236cb8a0491770c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.796555 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d331a13d13f34456be057dd9cc9fc23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.797874 1159 tablet_bootstrap.cc:654] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.799729 752 tablet_service.cc:1468] Processing CreateTablet for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 (DEFAULT_TABLE table=default.Table_3730b225150645e09a3a9b396cb134a2 [id=3e08d37ea03941308fbbcf7ad05a88c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.800019 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.782071 1016 tablet_service.cc:1468] Processing CreateTablet for tablet eb74a142c9974d108e0830b634bf4c16 (DEFAULT_TABLE table=default.Table_ae6c196463434b00a4de0e1349115428 [id=9a29094221c14561b3fab8f35c50b915]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.800305 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb74a142c9974d108e0830b634bf4c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.800679 1155 tablet_bootstrap.cc:654] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.782094 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 25a6407044644188922f727d2a01de8e (DEFAULT_TABLE table=default.Table_56965d558fb9442fa7c189c772e55937 [id=1a2b3834e2294d909f399e980f12f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.803593 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a6407044644188922f727d2a01de8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.803793 755 tablet_service.cc:1468] Processing CreateTablet for tablet 10f26ebecfd24c2b9ad7a6fe4fead40b (DEFAULT_TABLE table=default.Table_44ab2296b2644a9282db2646084a4c95 [id=729f7fea16df434a9e8bdd9052b1e7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.804108 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f26ebecfd24c2b9ad7a6fe4fead40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.781663 1022 tablet_service.cc:1468] Processing CreateTablet for tablet c2f0c1c26c49457db605222093b367cf (DEFAULT_TABLE table=default.Table_8ccb29231d6746ab93412d34b939117f [id=9bad87286c4a4850b11d8f49a8a46d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.804592 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f0c1c26c49457db605222093b367cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.781616 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 (DEFAULT_TABLE table=default.Table_3730b225150645e09a3a9b396cb134a2 [id=3e08d37ea03941308fbbcf7ad05a88c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.808327 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.810739 757 tablet_service.cc:1468] Processing CreateTablet for tablet 0c9c652531df452c963691ff3ac5fb7e (DEFAULT_TABLE table=default.Table_704f6c68c7484e5582a42cef6e942588 [id=c746a9e9c88844909d8409b125863b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.810978 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9c652531df452c963691ff3ac5fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.811172 1155 tablet_bootstrap.cc:492] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.811267 1155 ts_tablet_manager.cc:1397] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.003s
I20250629 01:52:01.811806 1155 raft_consensus.cc:357] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.811923 1155 raft_consensus.cc:383] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.811959 1155 raft_consensus.cc:738] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.781778 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 10f26ebecfd24c2b9ad7a6fe4fead40b (DEFAULT_TABLE table=default.Table_44ab2296b2644a9282db2646084a4c95 [id=729f7fea16df434a9e8bdd9052b1e7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.812114 1155 consensus_queue.cc:260] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.812237 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f26ebecfd24c2b9ad7a6fe4fead40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.814126 2526 ts_tablet_manager.cc:1918] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.814196 2526 log.cc:1199] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/68cedbd093b4420d8f7b29219ce8f77b
I20250629 01:52:01.814421 2526 ts_tablet_manager.cc:1939] T 68cedbd093b4420d8f7b29219ce8f77b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.814715 1155 ts_tablet_manager.cc:1428] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:01.814783 2526 tablet_replica.cc:331] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.814915 2526 raft_consensus.cc:2241] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.814924 1155 tablet_bootstrap.cc:492] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.815011 2526 raft_consensus.cc:2270] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.815083 745 tablet_service.cc:1468] Processing CreateTablet for tablet 81e67d94ba304f9eb030b854fc17d0d7 (DEFAULT_TABLE table=default.Table_e033aedd05ca4aef91836f468c2adfb0 [id=f74d9b675ea940259a17ba01881d3292]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.815303 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e67d94ba304f9eb030b854fc17d0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.815531 2526 ts_tablet_manager.cc:1905] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.816967 1155 tablet_bootstrap.cc:654] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.820122 748 tablet_service.cc:1468] Processing CreateTablet for tablet 25a6407044644188922f727d2a01de8e (DEFAULT_TABLE table=default.Table_56965d558fb9442fa7c189c772e55937 [id=1a2b3834e2294d909f399e980f12f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.820271 2526 ts_tablet_manager.cc:1918] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.820385 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a6407044644188922f727d2a01de8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.781433 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 1966a23b1fe14e6a94bd24ee4e8447a0 (DEFAULT_TABLE table=default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6 [id=8f6f7d0a2f4f43b8a3163e41db7e7351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.820930 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1966a23b1fe14e6a94bd24ee4e8447a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.821048 2526 log.cc:1199] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f4b7d0a44bbe4d8c81e8d0c444dbb55b
I20250629 01:52:01.821305 2526 ts_tablet_manager.cc:1939] T f4b7d0a44bbe4d8c81e8d0c444dbb55b P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.821612 2526 tablet_replica.cc:331] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.821734 2526 raft_consensus.cc:2241] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.821827 2526 raft_consensus.cc:2270] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.824307 749 tablet_service.cc:1468] Processing CreateTablet for tablet 24f980dafd8243738236cb8a0491770c (DEFAULT_TABLE table=default.Table_3e8277c1c24c4c58aba9b1f245d490c6 [id=3e9b6b745e5c4911984bb347687410de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.824457 2526 ts_tablet_manager.cc:1905] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.824558 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f980dafd8243738236cb8a0491770c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.824857 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 8295e8178eff4d5eb95ac885092731e0 (table default.Table_0d8a47c941e64eb1bd5332c0dbee8504 [id=c17c3600bb2447d69097c8223ce973f3]) successfully deleted
W20250629 01:52:01.825045 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet ee7fb74854db4034a9d44f6d025e6756 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7fb74854db4034a9d44f6d025e6756 already in progress: creating tablet
I20250629 01:52:01.814939 746 tablet_service.cc:1468] Processing CreateTablet for tablet eb74a142c9974d108e0830b634bf4c16 (DEFAULT_TABLE table=default.Table_ae6c196463434b00a4de0e1349115428 [id=9a29094221c14561b3fab8f35c50b915]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.828943 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb74a142c9974d108e0830b634bf4c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.829108 2526 ts_tablet_manager.cc:1918] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.829187 2526 log.cc:1199] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/74b3badd684d45fa88dc8cbcfc61001a
I20250629 01:52:01.829420 2526 ts_tablet_manager.cc:1939] T 74b3badd684d45fa88dc8cbcfc61001a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.832706 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1d2e2e2a1df5473392d9bc2d2ca9a993
I20250629 01:52:01.833209 1159 tablet_bootstrap.cc:492] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.833302 1159 ts_tablet_manager.cc:1397] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.000s
I20250629 01:52:01.833348 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9683e9406d2840bfbec404229f77ac50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.833775 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9683e9406d2840bfbec404229f77ac50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.833798 1159 raft_consensus.cc:357] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.833910 1159 raft_consensus.cc:383] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.833958 1159 raft_consensus.cc:738] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.834123 1159 consensus_queue.cc:260] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.834239 2526 tablet_replica.cc:331] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.834367 2526 raft_consensus.cc:2241] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.834767 2651 raft_consensus.cc:491] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.834865 2651 raft_consensus.cc:513] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.835247 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet bb98aa9d365541969ea1dc70960760b2
I20250629 01:52:01.835357 2651 leader_election.cc:290] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.837236 896 tablet_service.cc:1468] Processing CreateTablet for tablet 6a3f9c4dfce34596b5145cfeb887f727 (DEFAULT_TABLE table=default.Table_f3cd4aaddc584fb39053963cf9005c3d [id=5b2efc95946e4937b22e95b2fa4685e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.837525 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3f9c4dfce34596b5145cfeb887f727. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.837810 883 tablet_service.cc:1468] Processing CreateTablet for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 (DEFAULT_TABLE table=default.Table_3730b225150645e09a3a9b396cb134a2 [id=3e08d37ea03941308fbbcf7ad05a88c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.838081 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.838485 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3 (DEFAULT_TABLE table=default.Table_c24aa328eefc49d6979d74d3945cc219 [id=357d8ab96405410fbd77680046a87e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.838615 880 tablet_service.cc:1468] Processing CreateTablet for tablet 1966a23b1fe14e6a94bd24ee4e8447a0 (DEFAULT_TABLE table=default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6 [id=8f6f7d0a2f4f43b8a3163e41db7e7351]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.838735 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.838868 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1966a23b1fe14e6a94bd24ee4e8447a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.839083 881 tablet_service.cc:1468] Processing CreateTablet for tablet 8688bcb3523f4a7fa9366f9b648274e8 (DEFAULT_TABLE table=default.Table_b614c356b8dd4e5d854d84b62f1ff989 [id=e2afb5601a81486891b231f84ea74239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.839332 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8688bcb3523f4a7fa9366f9b648274e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.840265 877 tablet_service.cc:1468] Processing CreateTablet for tablet 8965a607d2a141ec94d9fdbf605f4bd7 (DEFAULT_TABLE table=default.Table_3e0a0e61a2314bdba910cdd875e456bf [id=e77821f698bd41d5ad96c03c31c58f5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.840422 885 tablet_service.cc:1468] Processing CreateTablet for tablet 10f26ebecfd24c2b9ad7a6fe4fead40b (DEFAULT_TABLE table=default.Table_44ab2296b2644a9282db2646084a4c95 [id=729f7fea16df434a9e8bdd9052b1e7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.840507 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8965a607d2a141ec94d9fdbf605f4bd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.840668 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10f26ebecfd24c2b9ad7a6fe4fead40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.840843 892 tablet_service.cc:1468] Processing CreateTablet for tablet d331a13d13f34456be057dd9cc9fc23b (DEFAULT_TABLE table=default.Table_dc29904c04dd44f9b8f1dc8bdf527229 [id=0c63248009934926aa42aace97554161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.841075 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d331a13d13f34456be057dd9cc9fc23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.842038 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ee1e7985f4b4422ba607962bf1fdc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.842291 888 tablet_service.cc:1468] Processing CreateTablet for tablet eb74a142c9974d108e0830b634bf4c16 (DEFAULT_TABLE table=default.Table_ae6c196463434b00a4de0e1349115428 [id=9a29094221c14561b3fab8f35c50b915]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.842530 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb74a142c9974d108e0830b634bf4c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.843058 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 40592f11f430469f9c8e82eba25b38ed (table default.Table_8e4f4f0655b44c68953073e4f0298fb8 [id=c0e46a0b48ee4ae58b819cd01a157a47]) successfully deleted
I20250629 01:52:01.843202 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 68cedbd093b4420d8f7b29219ce8f77b (table default.Table_31aef3786c3841c096a18ca7abfa40df [id=b46cf5dd7d524d0cb23f236b318eafca]) successfully deleted
I20250629 01:52:01.843343 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f4b7d0a44bbe4d8c81e8d0c444dbb55b (table default.Table_2acf6246904d41ab94aad9dec7df2291 [id=df90a81df72e4bc09675d9b0203888fc]) successfully deleted
I20250629 01:52:01.843487 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 74b3badd684d45fa88dc8cbcfc61001a (table default.Table_75f56cc1109c41f2a0a3ba83902b88ea [id=fcdd225098994f2f8c425a8fc3165d24]) successfully deleted
I20250629 01:52:01.844269 895 tablet_service.cc:1468] Processing CreateTablet for tablet 25a6407044644188922f727d2a01de8e (DEFAULT_TABLE table=default.Table_56965d558fb9442fa7c189c772e55937 [id=1a2b3834e2294d909f399e980f12f679]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.844300 878 tablet_service.cc:1468] Processing CreateTablet for tablet 0c9c652531df452c963691ff3ac5fb7e (DEFAULT_TABLE table=default.Table_704f6c68c7484e5582a42cef6e942588 [id=c746a9e9c88844909d8409b125863b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.844482 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.844523 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a6407044644188922f727d2a01de8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.844619 882 tablet_service.cc:1515] Processing DeleteTablet for tablet bb98aa9d365541969ea1dc70960760b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.846542 891 tablet_service.cc:1468] Processing CreateTablet for tablet 81e67d94ba304f9eb030b854fc17d0d7 (DEFAULT_TABLE table=default.Table_e033aedd05ca4aef91836f468c2adfb0 [id=f74d9b675ea940259a17ba01881d3292]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.848275 879 tablet_service.cc:1468] Processing CreateTablet for tablet 24f980dafd8243738236cb8a0491770c (DEFAULT_TABLE table=default.Table_3e8277c1c24c4c58aba9b1f245d490c6 [id=3e9b6b745e5c4911984bb347687410de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.848495 893 tablet_service.cc:1515] Processing DeleteTablet for tablet e45d1c8068e346039524312a058e9de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.840278 886 tablet_service.cc:1468] Processing CreateTablet for tablet c2f0c1c26c49457db605222093b367cf (DEFAULT_TABLE table=default.Table_8ccb29231d6746ab93412d34b939117f [id=9bad87286c4a4850b11d8f49a8a46d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.849261 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e67d94ba304f9eb030b854fc17d0d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.849265 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0c422ceed7594a18b32e5f70350e5b7b
I20250629 01:52:01.849624 450 catalog_manager.cc:6652] ReplaceTablet: tablet 0c422ceed7594a18b32e5f70350e5b7b deleted and replaced by tablet c2e36e81d5d14fecb5bed176c6fc996d
I20250629 01:52:01.849879 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0c1c7d79b0a74e8ea1c1b1f5036283dd" } new_table_name: "default.Table_19bddb7e84fa4bffba988e2b637d6728" modify_external_catalogs: true
I20250629 01:52:01.851008 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 78e379d748434664b4e2684d3f6012ab
I20250629 01:52:01.852031 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5b6cb001443c433a8a3b15de2a24fd9d" } new_table_name: "default.Table_83341ece52744f4e90689906ee72b37a" modify_external_catalogs: true
I20250629 01:52:01.852880 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aa8dfd712c043ad9f8ff56b064bb8fd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:01.853032 1052 raft_consensus.cc:2466] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:01.853925 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_730ee6fd4aca4315aaf72504cecdbad5" } new_table_name: "default.Table_b2a6df4f43c2485896340d3cf021d2f8" modify_external_catalogs: true
I20250629 01:52:01.854759 710 leader_election.cc:304] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.854959 2651 raft_consensus.cc:2802] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:01.855010 2651 raft_consensus.cc:491] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:01.855057 2651 raft_consensus.cc:3058] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:01.855906 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.856046 744 tablet_service.cc:1515] Processing DeleteTablet for tablet bb98aa9d365541969ea1dc70960760b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.856155 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c422ceed7594a18b32e5f70350e5b7b with delete_type TABLET_DATA_DELETED (replaced by tablet c2e36e81d5d14fecb5bed176c6fc996d) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.856278 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 78e379d748434664b4e2684d3f6012ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.857216 2651 raft_consensus.cc:513] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.857671 2651 leader_election.cc:290] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.862239 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet e45d1c8068e346039524312a058e9de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.862449 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 78e379d748434664b4e2684d3f6012ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.862479 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_13401ad7d0e2429a8e31aff8e7a61a3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.862596 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.862825 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet bb98aa9d365541969ea1dc70960760b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:01.862850 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13401ad7d0e2429a8e31aff8e7a61a3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.863003 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c422ceed7594a18b32e5f70350e5b7b with delete_type TABLET_DATA_DELETED (replaced by tablet c2e36e81d5d14fecb5bed176c6fc996d) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:01.784071 1154 tablet_bootstrap.cc:492] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.863204 1154 ts_tablet_manager.cc:1397] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20250629 01:52:01.863967 1154 raft_consensus.cc:357] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.864156 1154 raft_consensus.cc:383] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.864231 1154 raft_consensus.cc:738] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.864413 1154 consensus_queue.cc:260] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.865015 1154 ts_tablet_manager.cc:1428] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:01.866745 452 master_service.cc:948] ReplaceTablet: received request to replace tablet b6fa908ded8240bb962674e1efb29c74 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:01.867275 1154 tablet_bootstrap.cc:492] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.867590 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c422ceed7594a18b32e5f70350e5b7b with delete_type TABLET_DATA_DELETED (replaced by tablet c2e36e81d5d14fecb5bed176c6fc996d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.867807 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 78e379d748434664b4e2684d3f6012ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.868431 1154 tablet_bootstrap.cc:654] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.875665 888 tablet_service.cc:1468] Processing CreateTablet for tablet c4deb1f0bbd04c3e8b9d35b30f2b5307 (DEFAULT_TABLE table=default.Table_1e4a069a8f0a46d0a3f9da0bb78914be [id=79c5712b02674ab59a3899352868bbf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.875706 877 tablet_service.cc:1468] Processing CreateTablet for tablet 229ab5b547634fb59089510905e49a3e (DEFAULT_TABLE table=default.Table_0c86a89fdf674dfc856f5a107daad436 [id=af0787145f544902be2d758b2b957d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.875928 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4deb1f0bbd04c3e8b9d35b30f2b5307. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.876145 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229ab5b547634fb59089510905e49a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.876390 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aa8dfd712c043ad9f8ff56b064bb8fd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:01.876516 1052 raft_consensus.cc:3058] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:01.876873 710 consensus_peers.cc:489] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50d23f589de84dd5b150a17cee00c411. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:01.878685 1052 raft_consensus.cc:2466] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:01.880903 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2de995f275184ec9a2ac43aa496d9cad consensus state has no leader
I20250629 01:52:01.883735 740 tablet_service.cc:1468] Processing CreateTablet for tablet 8688bcb3523f4a7fa9366f9b648274e8 (DEFAULT_TABLE table=default.Table_b614c356b8dd4e5d854d84b62f1ff989 [id=e2afb5601a81486891b231f84ea74239]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.883736 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_13e1b0a3c2bc48b8ae300865cb64d7b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:01.883885 756 tablet_service.cc:1468] Processing CreateTablet for tablet 229ab5b547634fb59089510905e49a3e (DEFAULT_TABLE table=default.Table_0c86a89fdf674dfc856f5a107daad436 [id=af0787145f544902be2d758b2b957d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.884016 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8688bcb3523f4a7fa9366f9b648274e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:01.884128 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13e1b0a3c2bc48b8ae300865cb64d7b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.884183 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229ab5b547634fb59089510905e49a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.884279 744 tablet_service.cc:1468] Processing CreateTablet for tablet c4deb1f0bbd04c3e8b9d35b30f2b5307 (DEFAULT_TABLE table=default.Table_1e4a069a8f0a46d0a3f9da0bb78914be [id=79c5712b02674ab59a3899352868bbf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.884511 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4deb1f0bbd04c3e8b9d35b30f2b5307. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.885046 883 tablet_service.cc:1468] Processing CreateTablet for tablet 059b0999c6114f4681be7b97cd9eb071 (DEFAULT_TABLE table=default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.885285 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059b0999c6114f4681be7b97cd9eb071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.889484 892 tablet_service.cc:1468] Processing CreateTablet for tablet a6ace901512449f8b93281b8b1d69dde (DEFAULT_TABLE table=default.Table_b841397a6b4b40b9ae05c1aa6d69cda1 [id=5d74773751e44e069d953c374259e5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.889750 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ace901512449f8b93281b8b1d69dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.890287 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c4deb1f0bbd04c3e8b9d35b30f2b5307 (DEFAULT_TABLE table=default.Table_1e4a069a8f0a46d0a3f9da0bb78914be [id=79c5712b02674ab59a3899352868bbf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.890496 710 leader_election.cc:304] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:01.890591 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4deb1f0bbd04c3e8b9d35b30f2b5307. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.890872 2534 raft_consensus.cc:2802] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:01.890955 2534 raft_consensus.cc:695] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:01.891119 2534 consensus_queue.cc:237] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:01.892287 743 tablet_service.cc:1468] Processing CreateTablet for tablet a6ace901512449f8b93281b8b1d69dde (DEFAULT_TABLE table=default.Table_b841397a6b4b40b9ae05c1aa6d69cda1 [id=5d74773751e44e069d953c374259e5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.892554 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ace901512449f8b93281b8b1d69dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.895385 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 059b0999c6114f4681be7b97cd9eb071 (DEFAULT_TABLE table=default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.895628 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059b0999c6114f4681be7b97cd9eb071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.895726 885 tablet_service.cc:1468] Processing CreateTablet for tablet 290b433bc4054f7e87c44884892e76d6 (DEFAULT_TABLE table=default.Table_90ea22302d294dd48870c71a89b49025 [id=c5e8964604654e2085d4cd7e50c70a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.895990 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290b433bc4054f7e87c44884892e76d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.896279 738 tablet_service.cc:1468] Processing CreateTablet for tablet 059b0999c6114f4681be7b97cd9eb071 (DEFAULT_TABLE table=default.Table_af7bb496140a4999b9999124452b40cd [id=2b6ac9dc82a64445a8f9af84cf5b8c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.896522 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 059b0999c6114f4681be7b97cd9eb071. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.900287 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a6ace901512449f8b93281b8b1d69dde (DEFAULT_TABLE table=default.Table_b841397a6b4b40b9ae05c1aa6d69cda1 [id=5d74773751e44e069d953c374259e5a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.900596 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6ace901512449f8b93281b8b1d69dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.903714 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 229ab5b547634fb59089510905e49a3e (DEFAULT_TABLE table=default.Table_0c86a89fdf674dfc856f5a107daad436 [id=af0787145f544902be2d758b2b957d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.904008 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 229ab5b547634fb59089510905e49a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.903777 892 tablet_service.cc:1468] Processing CreateTablet for tablet 07ef448e4fda4091bd995448b352e6a2 (DEFAULT_TABLE table=default.Table_830b38557e8d4844ae1ab213c459309b [id=ba5059cf02504dbdaea2790efeecb30d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.904363 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07ef448e4fda4091bd995448b352e6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.907281 2762 raft_consensus.cc:491] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.907397 2762 raft_consensus.cc:513] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.907812 2526 raft_consensus.cc:2270] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.907891 2762 leader_election.cc:290] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.908193 1159 ts_tablet_manager.cc:1428] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.075s user 0.002s sys 0.000s
I20250629 01:52:01.908264 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec0ca04e5f6b42df8567deb53a3e5f53" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.908474 1159 tablet_bootstrap.cc:492] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.908511 2526 ts_tablet_manager.cc:1905] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:01.908708 988 leader_election.cc:343] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.909698 1159 tablet_bootstrap.cc:654] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.911558 1158 raft_consensus.cc:2241] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.911710 1158 raft_consensus.cc:2270] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.912550 1158 ts_tablet_manager.cc:1905] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.912834 888 tablet_service.cc:1468] Processing CreateTablet for tablet 5064e7b3d4d14f868e9aba9c3175382d (DEFAULT_TABLE table=default.Table_5ef79ed619d54eb28e78246a1a4f2b0f [id=4a034d3ca54540a9b22fce35c0666126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.913132 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5064e7b3d4d14f868e9aba9c3175382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.913908 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f980dafd8243738236cb8a0491770c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.914160 2670 raft_consensus.cc:491] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.914256 2670 raft_consensus.cc:513] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.914309 2526 ts_tablet_manager.cc:1918] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.914403 2526 log.cc:1199] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e3b94f8250a143b096703f2b0386e3cd
I20250629 01:52:01.914669 2526 ts_tablet_manager.cc:1939] T e3b94f8250a143b096703f2b0386e3cd P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.914757 2670 leader_election.cc:290] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.915035 2526 tablet_replica.cc:331] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:01.915052 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet e3b94f8250a143b096703f2b0386e3cd (table default.Table_b19a6c8eeb7f4f9b85050ca2555cf43d [id=2712c05036474b1a9be32b24c00e3dee]) successfully deleted
I20250629 01:52:01.915180 2526 raft_consensus.cc:2241] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.915284 2526 raft_consensus.cc:2270] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.916193 756 tablet_service.cc:1468] Processing CreateTablet for tablet 5064e7b3d4d14f868e9aba9c3175382d (DEFAULT_TABLE table=default.Table_5ef79ed619d54eb28e78246a1a4f2b0f [id=4a034d3ca54540a9b22fce35c0666126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.916302 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f0c1c26c49457db605222093b367cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.916486 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5064e7b3d4d14f868e9aba9c3175382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.917847 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 5064e7b3d4d14f868e9aba9c3175382d (DEFAULT_TABLE table=default.Table_5ef79ed619d54eb28e78246a1a4f2b0f [id=4a034d3ca54540a9b22fce35c0666126]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.918287 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5064e7b3d4d14f868e9aba9c3175382d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.919523 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 07ef448e4fda4091bd995448b352e6a2 (DEFAULT_TABLE table=default.Table_830b38557e8d4844ae1ab213c459309b [id=ba5059cf02504dbdaea2790efeecb30d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.919986 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c9c652531df452c963691ff3ac5fb7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.921850 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07ef448e4fda4091bd995448b352e6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.921960 2526 ts_tablet_manager.cc:1905] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.926946 2526 ts_tablet_manager.cc:1918] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:01.927032 2526 log.cc:1199] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/258994dd24804f4aaa5e299436fac9b0
I20250629 01:52:01.927256 2526 ts_tablet_manager.cc:1939] T 258994dd24804f4aaa5e299436fac9b0 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.927776 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 258994dd24804f4aaa5e299436fac9b0 (table default.Table_a8bede9e061941eaabacba34109b16ab [id=c4001552f202478e92afae5c3aacfa9b]) successfully deleted
W20250629 01:52:01.927975 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4ee1e7985f4b4422ba607962bf1fdc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ee1e7985f4b4422ba607962bf1fdc42 already in progress: creating tablet
W20250629 01:52:01.928174 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 already in progress: creating tablet
W20250629 01:52:01.928326 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet bb98aa9d365541969ea1dc70960760b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb98aa9d365541969ea1dc70960760b2 already in progress: creating tablet
W20250629 01:52:01.928462 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet e45d1c8068e346039524312a058e9de7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45d1c8068e346039524312a058e9de7 already in progress: creating tablet
W20250629 01:52:01.928606 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 0c422ceed7594a18b32e5f70350e5b7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c422ceed7594a18b32e5f70350e5b7b already in progress: creating tablet
W20250629 01:52:01.928781 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 78e379d748434664b4e2684d3f6012ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e379d748434664b4e2684d3f6012ab already in progress: creating tablet
I20250629 01:52:01.929462 1155 tablet_bootstrap.cc:492] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.929594 1155 ts_tablet_manager.cc:1397] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.115s user 0.002s sys 0.002s
I20250629 01:52:01.930513 1155 raft_consensus.cc:357] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.930681 1155 raft_consensus.cc:383] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.931481 1155 raft_consensus.cc:738] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.931741 1155 consensus_queue.cc:260] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.931955 745 tablet_service.cc:1515] Processing DeleteTablet for tablet e45d1c8068e346039524312a058e9de7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.932339 1155 ts_tablet_manager.cc:1428] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:01.932297 884 tablet_service.cc:1468] Processing CreateTablet for tablet 9370ec79715e4e5fa4766f202765f04b (DEFAULT_TABLE table=default.Table_55565d53ffdd403595e1e5cb1fd7effb [id=a1aced4f425d4711bdea9093f19c791c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.932569 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9370ec79715e4e5fa4766f202765f04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.932626 1155 tablet_bootstrap.cc:492] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:01.932713 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 7908afa2abaa4521b44f954e0790f986 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.933825 747 tablet_service.cc:1468] Processing CreateTablet for tablet 07ef448e4fda4091bd995448b352e6a2 (DEFAULT_TABLE table=default.Table_830b38557e8d4844ae1ab213c459309b [id=ba5059cf02504dbdaea2790efeecb30d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.937837 879 tablet_service.cc:1468] Processing CreateTablet for tablet d2bd46f64051455a8e7fb306b5d0c810 (DEFAULT_TABLE table=default.Table_952ee4fb7e584b3cb71d943396d62c65 [id=2a1a5354c72740c09618b28963a01626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.934062 1155 tablet_bootstrap.cc:654] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.938174 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07ef448e4fda4091bd995448b352e6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.938966 449 catalog_manager.cc:5582] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:01.939599 1031 tablet_service.cc:1468] Processing CreateTablet for tablet d331a13d13f34456be057dd9cc9fc23b (DEFAULT_TABLE table=default.Table_dc29904c04dd44f9b8f1dc8bdf527229 [id=0c63248009934926aa42aace97554161]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.941092 1154 tablet_bootstrap.cc:492] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.939945 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e0727102d54294a7c13a4aeef311e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:01.941200 1154 ts_tablet_manager.cc:1397] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.000s
I20250629 01:52:01.934696 1159 tablet_bootstrap.cc:492] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:01.941730 1159 ts_tablet_manager.cc:1397] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250629 01:52:01.941736 1154 raft_consensus.cc:357] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.941861 1154 raft_consensus.cc:383] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.941897 1154 raft_consensus.cc:738] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.934165 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0f835b3d78af49359eb2e2c8dc70951b (DEFAULT_TABLE table=default.Table_0a2224d72ed24f8f938582dfde5398dc [id=675bc5684fc34b39bf7d8f616995854b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.939599 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aa8dfd712c043ad9f8ff56b064bb8fd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.942057 1154 consensus_queue.cc:260] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.940266 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aa8dfd712c043ad9f8ff56b064bb8fd" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:01.942406 1154 ts_tablet_manager.cc:1428] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:01.942467 1159 raft_consensus.cc:357] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.942595 1159 raft_consensus.cc:383] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.942610 1154 tablet_bootstrap.cc:492] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.943591 1154 tablet_bootstrap.cc:654] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.943820 1159 raft_consensus.cc:738] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.937958 892 tablet_service.cc:1468] Processing CreateTablet for tablet 15a01d127e0a49bcbc37dbb95bca4d0d (DEFAULT_TABLE table=default.Table_81bfc89e69ed457089280d6e2049316e [id=ad032f21aad54836b16213d7417e15bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.944350 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:01.944623 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:01.944890 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:01.941781 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2bd46f64051455a8e7fb306b5d0c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.943626 1155 tablet_bootstrap.cc:492] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:01.946111 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a01d127e0a49bcbc37dbb95bca4d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.946887 1155 ts_tablet_manager.cc:1397] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250629 01:52:01.944890 1159 consensus_queue.cc:260] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:01.947760 712 consensus_peers.cc:489] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a792a26725954adea15ad57e31bfb224. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:01.952421 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bbb61a515a4744c18d21d1b96f012f7b consensus state has no leader
I20250629 01:52:01.952756 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2ebb3eb6a01f4d388ddaa95cf3cfad0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.953223 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ebb3eb6a01f4d388ddaa95cf3cfad0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:01.954741 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60f427fa19be4b3381ed9ae951cb842a consensus state has no leader
I20250629 01:52:01.955029 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_70cddd5986ff4223b20b1e8d07c63456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.955469 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70cddd5986ff4223b20b1e8d07c63456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:01.957891 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3f88c1cdd331456683f3fe4d95aa18af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.958272 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f88c1cdd331456683f3fe4d95aa18af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:01.959334 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cf193900a2842e4983a00c8a4f301a9 consensus state has no leader
I20250629 01:52:01.959583 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6beeaafab09745f49b66734e77f01269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:01.959920 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6beeaafab09745f49b66734e77f01269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:01.961352 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 47972.5us
I20250629 01:52:01.961923 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet b6fa908ded8240bb962674e1efb29c74
I20250629 01:52:01.962023 1154 tablet_bootstrap.cc:492] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.962106 1154 ts_tablet_manager.cc:1397] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250629 01:52:01.962262 452 catalog_manager.cc:6652] ReplaceTablet: tablet b6fa908ded8240bb962674e1efb29c74 deleted and replaced by tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec
I20250629 01:52:01.962649 1154 raft_consensus.cc:357] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.962781 1154 raft_consensus.cc:383] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.962831 1154 raft_consensus.cc:738] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.962975 1154 consensus_queue.cc:260] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.963374 1154 ts_tablet_manager.cc:1428] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250629 01:52:01.963599 1154 tablet_bootstrap.cc:492] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.964829 2762 raft_consensus.cc:491] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.964883 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 290b433bc4054f7e87c44884892e76d6 (DEFAULT_TABLE table=default.Table_90ea22302d294dd48870c71a89b49025 [id=c5e8964604654e2085d4cd7e50c70a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.964919 2762 raft_consensus.cc:513] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.965111 753 tablet_service.cc:1468] Processing CreateTablet for tablet 15a01d127e0a49bcbc37dbb95bca4d0d (DEFAULT_TABLE table=default.Table_81bfc89e69ed457089280d6e2049316e [id=ad032f21aad54836b16213d7417e15bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.965350 2762 leader_election.cc:290] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.965389 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a01d127e0a49bcbc37dbb95bca4d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.966225 1154 tablet_bootstrap.cc:654] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.966459 2762 raft_consensus.cc:491] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.966554 2762 raft_consensus.cc:513] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.967038 2762 leader_election.cc:290] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.968394 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d2bd46f64051455a8e7fb306b5d0c810 (DEFAULT_TABLE table=default.Table_952ee4fb7e584b3cb71d943396d62c65 [id=2a1a5354c72740c09618b28963a01626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.968600 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 15a01d127e0a49bcbc37dbb95bca4d0d (DEFAULT_TABLE table=default.Table_81bfc89e69ed457089280d6e2049316e [id=ad032f21aad54836b16213d7417e15bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.968895 742 tablet_service.cc:1468] Processing CreateTablet for tablet 290b433bc4054f7e87c44884892e76d6 (DEFAULT_TABLE table=default.Table_90ea22302d294dd48870c71a89b49025 [id=c5e8964604654e2085d4cd7e50c70a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.969135 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290b433bc4054f7e87c44884892e76d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.972302 752 tablet_service.cc:1468] Processing CreateTablet for tablet d2bd46f64051455a8e7fb306b5d0c810 (DEFAULT_TABLE table=default.Table_952ee4fb7e584b3cb71d943396d62c65 [id=2a1a5354c72740c09618b28963a01626]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:01.972563 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2bd46f64051455a8e7fb306b5d0c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.972971 1159 ts_tablet_manager.cc:1428] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20250629 01:52:01.973184 1159 tablet_bootstrap.cc:492] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:01.975498 1154 tablet_bootstrap.cc:492] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:01.975610 1154 ts_tablet_manager.cc:1397] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
W20250629 01:52:01.948061 850 leader_election.cc:343] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: ac9d77381f7e4241aa546eaea3077168
I20250629 01:52:01.976640 1154 raft_consensus.cc:357] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.976786 1154 raft_consensus.cc:383] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:01.976758 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "476a6cf28db04f67bc23ddfc5cacad87" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.976846 1154 raft_consensus.cc:738] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:01.977005 1154 consensus_queue.cc:260] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.977145 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a582d4f907f340179024c0377d3893e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:01.977433 1154 ts_tablet_manager.cc:1428] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:01.977674 1154 tablet_bootstrap.cc:492] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:01.977968 888 tablet_service.cc:1515] Processing DeleteTablet for tablet b6fa908ded8240bb962674e1efb29c74 with delete_type TABLET_DATA_DELETED (replaced by tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:01.978158 2526 tablet_replica.cc:331] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:52:01.978139 988 leader_election.cc:343] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:01.978281 988 leader_election.cc:343] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.978324 2526 raft_consensus.cc:2241] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:01.963663 2670 raft_consensus.cc:491] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.978464 2526 raft_consensus.cc:2270] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:01.978554 2670 raft_consensus.cc:513] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.978899 2526 ts_tablet_manager.cc:1905] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:01.979089 2670 leader_election.cc:290] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:01.948284 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f835b3d78af49359eb2e2c8dc70951b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.985168 2526 ts_tablet_manager.cc:1918] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:01.985275 2526 log.cc:1199] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/b6fa908ded8240bb962674e1efb29c74
I20250629 01:52:01.985627 2526 ts_tablet_manager.cc:1939] T b6fa908ded8240bb962674e1efb29c74 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:01.986160 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15a01d127e0a49bcbc37dbb95bca4d0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:01.986622 712 consensus_peers.cc:489] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:01.986790 712 leader_election.cc:343] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:01.987078 712 leader_election.cc:343] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.987495 2548 raft_consensus.cc:491] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.988075 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]) successfully deleted
I20250629 01:52:01.988535 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2bd46f64051455a8e7fb306b5d0c810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.988140 2548 raft_consensus.cc:513] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.989025 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 290b433bc4054f7e87c44884892e76d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.989120 2548 leader_election.cc:290] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.990695 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43b845c3e7174618bd5c394d7206a634" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:01.991714 710 leader_election.cc:343] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:01.993414 1159 tablet_bootstrap.cc:654] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.995190 1154 tablet_bootstrap.cc:654] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:01.995420 2527 raft_consensus.cc:491] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:01.995615 2527 raft_consensus.cc:513] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:01.996207 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d331a13d13f34456be057dd9cc9fc23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:01.996383 2527 leader_election.cc:290] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:01.999920 1159 tablet_bootstrap.cc:492] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.000681 1159 ts_tablet_manager.cc:1397] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250629 01:52:02.000733 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.003252 1159 raft_consensus.cc:357] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:02.004597 1158 ts_tablet_manager.cc:1918] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.004695 1158 log.cc:1199] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/898b49812ece4b57a1a1015d9f26be2f
I20250629 01:52:02.004982 1158 ts_tablet_manager.cc:1939] T 898b49812ece4b57a1a1015d9f26be2f P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.005235 1154 tablet_bootstrap.cc:492] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.005342 1154 ts_tablet_manager.cc:1397] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20250629 01:52:02.005996 1154 raft_consensus.cc:357] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.006145 1154 raft_consensus.cc:383] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.006193 1154 raft_consensus.cc:738] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.006392 1159 raft_consensus.cc:383] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.006389 1154 consensus_queue.cc:260] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.007006 988 leader_election.cc:343] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.007365 1154 ts_tablet_manager.cc:1428] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:02.007683 1154 tablet_bootstrap.cc:492] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.008455 1158 tablet_replica.cc:331] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.008692 1158 raft_consensus.cc:2241] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.008855 1158 raft_consensus.cc:2270] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.009274 1154 tablet_bootstrap.cc:654] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.012627 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0f835b3d78af49359eb2e2c8dc70951b (DEFAULT_TABLE table=default.Table_0a2224d72ed24f8f938582dfde5398dc [id=675bc5684fc34b39bf7d8f616995854b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.013037 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f835b3d78af49359eb2e2c8dc70951b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.018748 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9370ec79715e4e5fa4766f202765f04b (DEFAULT_TABLE table=default.Table_55565d53ffdd403595e1e5cb1fd7effb [id=a1aced4f425d4711bdea9093f19c791c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.019058 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9370ec79715e4e5fa4766f202765f04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.009413 1158 ts_tablet_manager.cc:1905] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.021765 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0f835b3d78af49359eb2e2c8dc70951b (DEFAULT_TABLE table=default.Table_0a2224d72ed24f8f938582dfde5398dc [id=675bc5684fc34b39bf7d8f616995854b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.022076 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f835b3d78af49359eb2e2c8dc70951b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.022076 1154 tablet_bootstrap.cc:492] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.022275 1154 ts_tablet_manager.cc:1397] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20250629 01:52:02.022912 1154 raft_consensus.cc:357] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.023056 1154 raft_consensus.cc:383] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.023111 1154 raft_consensus.cc:738] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.023290 1154 consensus_queue.cc:260] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.023789 1154 ts_tablet_manager.cc:1428] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:02.024024 1154 tablet_bootstrap.cc:492] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.024150 1159 raft_consensus.cc:738] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.024658 2527 raft_consensus.cc:491] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.024613 1159 consensus_queue.cc:260] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:02.024765 2527 raft_consensus.cc:513] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.025210 2527 leader_election.cc:290] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.025647 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a8a9bc691e4b158276f34a51c0dc61" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:02.026068 988 leader_election.cc:343] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.026175 1159 ts_tablet_manager.cc:1428] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20250629 01:52:02.026403 1159 tablet_bootstrap.cc:492] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.026860 1154 tablet_bootstrap.cc:654] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.032138 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 898b49812ece4b57a1a1015d9f26be2f (table default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]) successfully deleted
I20250629 01:52:02.033017 756 tablet_service.cc:1515] Processing DeleteTablet for tablet b6fa908ded8240bb962674e1efb29c74 with delete_type TABLET_DATA_DELETED (replaced by tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.033871 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet b6fa908ded8240bb962674e1efb29c74 with delete_type TABLET_DATA_DELETED (replaced by tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.033758 1158 ts_tablet_manager.cc:1918] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.034106 1158 log.cc:1199] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/07e0727102d54294a7c13a4aeef311e1
I20250629 01:52:02.034400 1158 ts_tablet_manager.cc:1939] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.035028 1158 tablet_replica.cc:331] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.035238 1158 raft_consensus.cc:2241] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.035589 1158 raft_consensus.cc:2270] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.036046 1158 ts_tablet_manager.cc:1905] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.041128 1159 tablet_bootstrap.cc:654] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.041219 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9ef21c0a6bf74bd49725f221cb0b5b66" } new_table_name: "default.Table_fdb0c64a241c4ca9b76336e075b200bc" modify_external_catalogs: true
I20250629 01:52:02.048364 2651 raft_consensus.cc:491] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.048481 2651 raft_consensus.cc:513] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.049054 2651 leader_election.cc:290] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.049307 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ba5887dffc1460a89208447ecc6a198" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.049446 1052 raft_consensus.cc:2466] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:02.049800 710 leader_election.cc:304] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.050046 2651 raft_consensus.cc:2802] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.050113 2651 raft_consensus.cc:491] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.050170 2651 raft_consensus.cc:3058] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.050639 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3730b225150645e09a3a9b396cb134a2" } modify_external_catalogs: true
I20250629 01:52:02.050771 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3730b225150645e09a3a9b396cb134a2" } modify_external_catalogs: true
I20250629 01:52:02.051204 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a45ac1d6a41941178905d51452d4c4fc" } modify_external_catalogs: true
I20250629 01:52:02.051312 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a45ac1d6a41941178905d51452d4c4fc" } modify_external_catalogs: true
I20250629 01:52:02.052364 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55565d53ffdd403595e1e5cb1fd7effb" } modify_external_catalogs: true
I20250629 01:52:02.052563 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_55565d53ffdd403595e1e5cb1fd7effb" } modify_external_catalogs: true
I20250629 01:52:02.052637 2651 raft_consensus.cc:513] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.053094 2651 leader_election.cc:290] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.053565 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ba5887dffc1460a89208447ecc6a198" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:02.053725 1052 raft_consensus.cc:3058] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.054319 2670 raft_consensus.cc:491] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.054433 2670 raft_consensus.cc:513] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.054929 2670 leader_election.cc:290] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:02.055284 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43b845c3e7174618bd5c394d7206a634" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.055992 1052 raft_consensus.cc:2466] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:02.057731 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ba5887dffc1460a89208447ecc6a198" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.059343 1158 ts_tablet_manager.cc:1918] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.059444 1158 log.cc:1199] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/dc841eb171d04b118b1e705ceed4c861
I20250629 01:52:02.059707 1158 ts_tablet_manager.cc:1939] T dc841eb171d04b118b1e705ceed4c861 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.060412 710 leader_election.cc:304] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.060842 2548 raft_consensus.cc:2802] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:02.060931 2548 raft_consensus.cc:695] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:02.061125 2548 consensus_queue.cc:237] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.070432 1158 tablet_replica.cc:331] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.070595 1158 raft_consensus.cc:2241] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.070698 1158 raft_consensus.cc:2270] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.071738 1158 ts_tablet_manager.cc:1905] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.075224 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3ba5887dffc1460a89208447ecc6a198" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:02.080125 712 leader_election.cc:343] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.080255 712 leader_election.cc:304] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.080478 2651 raft_consensus.cc:2747] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:02.080718 850 leader_election.cc:343] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:52:02.081841 1158 ts_tablet_manager.cc:1918] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.081928 1158 log.cc:1199] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/7908afa2abaa4521b44f954e0790f986
I20250629 01:52:02.082162 1158 ts_tablet_manager.cc:1939] T 7908afa2abaa4521b44f954e0790f986 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.083149 1158 tablet_replica.cc:331] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.083300 1158 raft_consensus.cc:2241] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.083405 1158 raft_consensus.cc:2270] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.083827 1158 ts_tablet_manager.cc:1905] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.083886 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6d323c26638e4b2c9a01abd8feee7412" } modify_external_catalogs: true
I20250629 01:52:02.084010 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6d323c26638e4b2c9a01abd8feee7412" } modify_external_catalogs: true
W20250629 01:52:02.084128 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 50159.3us
I20250629 01:52:02.086443 1052 raft_consensus.cc:1273] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Refusing update from remote peer ef5c5979bb9e475f983af7c3e9e0e019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:52:02.086946 2548 consensus_queue.cc:1035] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:02.089704 1156 tablet_replica.cc:331] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:02.089923 1156 raft_consensus.cc:2241] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.089982 1158 ts_tablet_manager.cc:1918] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.090049 1156 raft_consensus.cc:2270] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.090085 1158 log.cc:1199] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4aa8dfd712c043ad9f8ff56b064bb8fd
I20250629 01:52:02.090325 1158 ts_tablet_manager.cc:1939] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.090554 1156 ts_tablet_manager.cc:1905] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.091001 1158 tablet_replica.cc:331] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.091145 1158 raft_consensus.cc:2241] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.091260 1158 raft_consensus.cc:2270] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.091647 1158 ts_tablet_manager.cc:1905] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.094705 1154 tablet_bootstrap.cc:492] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.094853 1154 ts_tablet_manager.cc:1397] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.003s
I20250629 01:52:02.095538 1154 raft_consensus.cc:357] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.095750 1154 raft_consensus.cc:383] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.095840 1154 raft_consensus.cc:738] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.096055 1154 consensus_queue.cc:260] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.096232 1158 ts_tablet_manager.cc:1918] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.096331 1158 log.cc:1199] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/476a6cf28db04f67bc23ddfc5cacad87
I20250629 01:52:02.096542 1158 ts_tablet_manager.cc:1939] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.096837 1154 ts_tablet_manager.cc:1428] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.097416 1154 tablet_bootstrap.cc:492] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.098515 2548 raft_consensus.cc:491] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.098666 2548 raft_consensus.cc:513] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.099610 710 leader_election.cc:343] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 476a6cf28db04f67bc23ddfc5cacad87
I20250629 01:52:02.099413 2548 leader_election.cc:290] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.096925 1158 tablet_replica.cc:331] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.099992 1158 raft_consensus.cc:2241] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.100066 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 07e0727102d54294a7c13a4aeef311e1 (table default.Table_a9fddce63afb42b4be73a8c4807e050a [id=d8e51d2596fa4215bd2e2a3c43c466f1]) successfully deleted
I20250629 01:52:02.100163 1158 raft_consensus.cc:2270] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.100354 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet dc841eb171d04b118b1e705ceed4c861 (table default.Table_7f95c64f736b4aee86abc0c7785ab8d6 [id=0429a74c16854b3092e37a247f33c497]) successfully deleted
I20250629 01:52:02.100620 1158 ts_tablet_manager.cc:1905] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.100610 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 7908afa2abaa4521b44f954e0790f986 (table default.Table_e998111141ca474dbff482e945a67465 [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]) successfully deleted
I20250629 01:52:02.100745 1159 tablet_bootstrap.cc:492] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
W20250629 01:52:02.100881 440 catalog_manager.cc:4895] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 327c303b6b9347f5b4cb78f74d41a44a because the tablet was not found. No further retry: Not found: Tablet not found: 327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:52:02.100914 1159 ts_tablet_manager.cc:1397] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.000s
I20250629 01:52:02.101037 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4aa8dfd712c043ad9f8ff56b064bb8fd (table default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]) successfully deleted
I20250629 01:52:02.101158 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 476a6cf28db04f67bc23ddfc5cacad87 (table default.Table_992098655a084dd1a8d522be5b7c1cc3 [id=b4d82748e8804c78a1bd48cef02e98a9]) successfully deleted
I20250629 01:52:02.101812 1159 raft_consensus.cc:357] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.102066 1159 raft_consensus.cc:383] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.102157 1159 raft_consensus.cc:738] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.102418 1159 consensus_queue.cc:260] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.103113 1159 ts_tablet_manager.cc:1428] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.103432 1159 tablet_bootstrap.cc:492] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.105039 1154 tablet_bootstrap.cc:654] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.105175 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_27543398d60941ce8b16591b00423498" } modify_external_catalogs: true
I20250629 01:52:02.105316 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_27543398d60941ce8b16591b00423498" } modify_external_catalogs: true
I20250629 01:52:02.106962 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_6f957d68e3dd4ba9a5bf56cc925385bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:02.107299 1159 tablet_bootstrap.cc:654] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:02.107388 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f957d68e3dd4ba9a5bf56cc925385bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.108431 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 72b2caf969db48458ea007ae03a0640f from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:02.108865 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_67461ab444b64f80854e45ddd5e4bcb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:02.109077 1158 ts_tablet_manager.cc:1918] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.109172 1158 log.cc:1199] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/689adaf0c7564d9ba0b31061e4e2d8cf
W20250629 01:52:02.109282 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67461ab444b64f80854e45ddd5e4bcb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.109406 1158 ts_tablet_manager.cc:1939] T 689adaf0c7564d9ba0b31061e4e2d8cf P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.110670 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a1da6e7dee904f23886f6d9d75c8e49f
I20250629 01:52:02.111372 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9ffe7a3207f44327bb5b618169fa2aaf" } new_table_name: "default.Table_07fa0e5cd5a045e4b0535093c0cb9ae9" modify_external_catalogs: true
I20250629 01:52:02.113411 1154 tablet_bootstrap.cc:492] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.113518 1154 ts_tablet_manager.cc:1397] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250629 01:52:02.113691 892 tablet_service.cc:1515] Processing DeleteTablet for tablet a1da6e7dee904f23886f6d9d75c8e49f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.114099 1154 raft_consensus.cc:357] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.114261 1154 raft_consensus.cc:383] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.114323 1154 raft_consensus.cc:738] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.114495 1154 consensus_queue.cc:260] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.114842 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 999dae3757ad4c57bf4e2a92ee5203ca (table default.Table_5d53a01d5be7418c988913852a7f91a3 [id=194bdf24776b480895832c6d00becb73], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 999dae3757ad4c57bf4e2a92ee5203ca consensus state has no leader
I20250629 01:52:02.116437 1154 ts_tablet_manager.cc:1428] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250629 01:52:02.118135 1154 tablet_bootstrap.cc:492] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.117825 1158 tablet_replica.cc:331] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.119484 1158 raft_consensus.cc:2241] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:52:02.119525 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38252fbce581492fbb3db848a2a31a49 consensus state has no leader
I20250629 01:52:02.119614 1158 raft_consensus.cc:2270] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:02.119596 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet a1da6e7dee904f23886f6d9d75c8e49f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6e7dee904f23886f6d9d75c8e49f already in progress: creating tablet
I20250629 01:52:02.120065 1159 tablet_bootstrap.cc:492] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.120533 1159 ts_tablet_manager.cc:1397] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20250629 01:52:02.120863 1158 ts_tablet_manager.cc:1905] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.121351 1159 raft_consensus.cc:357] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.121555 1159 raft_consensus.cc:383] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.121651 1159 raft_consensus.cc:738] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:52:02.121649 712 leader_election.cc:343] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:02.121975 712 leader_election.cc:343] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.121934 1159 consensus_queue.cc:260] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.122824 2651 raft_consensus.cc:491] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.122871 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "476a6cf28db04f67bc23ddfc5cacad87" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.122923 2651 raft_consensus.cc:513] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.123766 2651 leader_election.cc:290] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.122702 1154 tablet_bootstrap.cc:654] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.123953 1159 ts_tablet_manager.cc:1428] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:02.124256 1159 tablet_bootstrap.cc:492] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:02.124294 710 leader_election.cc:343] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 689adaf0c7564d9ba0b31061e4e2d8cf
I20250629 01:52:02.126792 1159 tablet_bootstrap.cc:654] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.130426 1154 tablet_bootstrap.cc:492] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.130527 1154 ts_tablet_manager.cc:1397] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
W20250629 01:52:02.133227 850 leader_election.cc:343] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 06e31520d3ad4c3395a59049755a4725
I20250629 01:52:02.133347 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "689adaf0c7564d9ba0b31061e4e2d8cf" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:02.133489 712 consensus_peers.cc:489] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:02.133591 712 leader_election.cc:343] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.133657 712 leader_election.cc:304] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.133847 2651 raft_consensus.cc:2747] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:02.134011 712 leader_election.cc:343] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.134097 712 leader_election.cc:304] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.134258 2651 raft_consensus.cc:2747] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.134944 1154 raft_consensus.cc:357] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.135073 1154 raft_consensus.cc:383] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.135125 1154 raft_consensus.cc:738] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.136056 1154 consensus_queue.cc:260] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.136826 1154 ts_tablet_manager.cc:1428] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250629 01:52:02.137055 1154 tablet_bootstrap.cc:492] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.137305 756 tablet_service.cc:1515] Processing DeleteTablet for tablet dc841eb171d04b118b1e705ceed4c861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.137768 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.138160 756 tablet_service.cc:1515] Processing DeleteTablet for tablet c5d45854191247bf8172d0bbfd01b6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.138629 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba5887dffc1460a89208447ecc6a198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.138944 1158 ts_tablet_manager.cc:1918] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.139031 1158 log.cc:1199] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ec0ca04e5f6b42df8567deb53a3e5f53
I20250629 01:52:02.139851 1158 ts_tablet_manager.cc:1939] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.140731 1159 tablet_bootstrap.cc:492] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.140774 1154 tablet_bootstrap.cc:654] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.140899 1159 ts_tablet_manager.cc:1397] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20250629 01:52:02.141718 1159 raft_consensus.cc:357] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.141942 1159 raft_consensus.cc:383] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.142038 1159 raft_consensus.cc:738] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.142288 1159 consensus_queue.cc:260] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.142640 2765 tablet.cc:1722] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:52:02.142940 1159 ts_tablet_manager.cc:1428] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.143227 1159 tablet_bootstrap.cc:492] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.144815 1159 tablet_bootstrap.cc:654] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.148746 1155 raft_consensus.cc:357] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.148900 1155 raft_consensus.cc:383] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.148954 1155 raft_consensus.cc:738] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.149257 1155 consensus_queue.cc:260] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.149857 1155 ts_tablet_manager.cc:1428] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.203s user 0.001s sys 0.001s
I20250629 01:52:02.150101 1155 tablet_bootstrap.cc:492] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.150895 1158 tablet_replica.cc:331] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.151065 1158 raft_consensus.cc:2241] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.152148 1158 raft_consensus.cc:2270] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.152576 1158 ts_tablet_manager.cc:1905] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.158581 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c64d8f4e6caa4ef4a4feb26802b0fbd8" } new_table_name: "default.Table_7a76965c683e460a8886557705100556" modify_external_catalogs: true
I20250629 01:52:02.169117 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.169745 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 9370ec79715e4e5fa4766f202765f04b (DEFAULT_TABLE table=default.Table_55565d53ffdd403595e1e5cb1fd7effb [id=a1aced4f425d4711bdea9093f19c791c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.170236 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9370ec79715e4e5fa4766f202765f04b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.172128 2791 tablet.cc:1722] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:52:02.174292 2534 raft_consensus.cc:491] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.174443 2534 raft_consensus.cc:513] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.175415 710 leader_election.cc:343] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 7908afa2abaa4521b44f954e0790f986
I20250629 01:52:02.175740 2534 leader_election.cc:290] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.180625 986 leader_election.cc:304] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:02.180910 986 raft_consensus.cc:2684] T 07e0727102d54294a7c13a4aeef311e1 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250629 01:52:02.184657 850 leader_election.cc:343] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ac9d77381f7e4241aa546eaea3077168
I20250629 01:52:02.184782 850 leader_election.cc:304] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.184996 2670 raft_consensus.cc:2747] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.185892 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 76118a6e684448b09f244d44c909ccaa (table default.Table_5794ea9989b54ab9bd7ecca30428c971 [id=dcc07a46945a4add8e667bfea51dfed7]). Expected version 5 got 0
I20250629 01:52:02.190301 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 689adaf0c7564d9ba0b31061e4e2d8cf (table default.Table_55cf597f7eab480299bb4d32ccf5868c [id=fb540507d98341b2801c8394decec635]) successfully deleted
I20250629 01:52:02.190563 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ec0ca04e5f6b42df8567deb53a3e5f53 (table default.Table_e72b0bd01cbb4f8eaa5caecb2d75133a [id=4a83041e3c7349f7a0b628c27e3a2139]) successfully deleted
W20250629 01:52:02.191012 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4ee1e7985f4b4422ba607962bf1fdc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ee1e7985f4b4422ba607962bf1fdc42 already in progress: creating tablet
I20250629 01:52:02.191313 1154 tablet_bootstrap.cc:492] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:52:02.191277 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet e45d1c8068e346039524312a058e9de7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45d1c8068e346039524312a058e9de7 already in progress: creating tablet
I20250629 01:52:02.191486 1154 ts_tablet_manager.cc:1397] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.001s
W20250629 01:52:02.191576 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 78e379d748434664b4e2684d3f6012ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e379d748434664b4e2684d3f6012ab already in progress: creating tablet
W20250629 01:52:02.191830 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 already in progress: creating tablet
W20250629 01:52:02.192077 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet bb98aa9d365541969ea1dc70960760b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb98aa9d365541969ea1dc70960760b2 already in progress: creating tablet
W20250629 01:52:02.192400 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 0c422ceed7594a18b32e5f70350e5b7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c422ceed7594a18b32e5f70350e5b7b already in progress: creating tablet
I20250629 01:52:02.192396 1154 raft_consensus.cc:357] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.192658 1154 raft_consensus.cc:383] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.192746 1154 raft_consensus.cc:738] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.192976 1154 consensus_queue.cc:260] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.193449 1154 ts_tablet_manager.cc:1428] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.193441 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b845c3e7174618bd5c394d7206a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.193681 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 898b49812ece4b57a1a1015d9f26be2f with delete_type TABLET_DATA_DELETED (replaced by tablet 573b329d020549a08c94891916164aeb) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.193742 1154 tablet_bootstrap.cc:492] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.193890 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a1da6e7dee904f23886f6d9d75c8e49f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.194401 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7908afa2abaa4521b44f954e0790f986" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.194936 2651 raft_consensus.cc:491] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.195040 2651 raft_consensus.cc:513] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.196421 710 consensus_peers.cc:489] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5d27d658892496a9b19b57949a8399a. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.197052 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5d45854191247bf8172d0bbfd01b6db" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.197088 2651 leader_election.cc:290] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.197221 1052 raft_consensus.cc:2466] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:02.197576 710 leader_election.cc:304] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.197897 2534 raft_consensus.cc:2802] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.198006 2534 raft_consensus.cc:491] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.198122 2534 raft_consensus.cc:3058] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.201269 2534 raft_consensus.cc:513] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.202059 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5d45854191247bf8172d0bbfd01b6db" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:02.202117 2534 leader_election.cc:290] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.202194 1159 tablet_bootstrap.cc:492] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.202240 1052 raft_consensus.cc:3058] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.202349 1159 ts_tablet_manager.cc:1397] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.007s
I20250629 01:52:02.203039 1154 tablet_bootstrap.cc:654] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.203004 1159 raft_consensus.cc:357] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.203223 1159 raft_consensus.cc:383] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.203302 1159 raft_consensus.cc:738] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.203507 1159 consensus_queue.cc:260] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.203995 1159 ts_tablet_manager.cc:1428] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:02.204512 1159 tablet_bootstrap.cc:492] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.204523 1052 raft_consensus.cc:2466] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:02.204859 710 leader_election.cc:304] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.205181 2534 raft_consensus.cc:2802] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:02.205396 2534 raft_consensus.cc:695] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:02.205610 2487 tablet.cc:1722] T 2fcead51fd9b43d0a52c4669a5e23fe5 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:02.205801 1159 tablet_bootstrap.cc:654] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.206118 2534 consensus_queue.cc:237] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.208153 1158 ts_tablet_manager.cc:1918] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:02.208272 1158 log.cc:1199] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/b6fa908ded8240bb962674e1efb29c74
I20250629 01:52:02.208619 1158 ts_tablet_manager.cc:1939] T b6fa908ded8240bb962674e1efb29c74 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.209483 2791 tablet.cc:1722] T 2fcead51fd9b43d0a52c4669a5e23fe5 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:02.216333 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 476a6cf28db04f67bc23ddfc5cacad87 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.222920 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6a9d2aa33a5c4d9ebecaa83c32b39dcb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:02.223044 775 raft_consensus.cc:3058] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.223779 1154 tablet_bootstrap.cc:492] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.223870 1154 ts_tablet_manager.cc:1397] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250629 01:52:02.224412 1154 raft_consensus.cc:357] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.224527 1154 raft_consensus.cc:383] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.224565 1154 raft_consensus.cc:738] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.224717 1154 consensus_queue.cc:260] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.225147 1154 ts_tablet_manager.cc:1428] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:02.225359 1154 tablet_bootstrap.cc:492] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.225410 775 raft_consensus.cc:2466] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:02.222141 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet a1da6e7dee904f23886f6d9d75c8e49f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.226790 1154 tablet_bootstrap.cc:654] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.228214 2670 raft_consensus.cc:491] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.228315 2670 raft_consensus.cc:513] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.228749 2670 leader_election.cc:290] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:02.229460 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec0ca04e5f6b42df8567deb53a3e5f53" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.231305 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "476a6cf28db04f67bc23ddfc5cacad87" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.231527 775 raft_consensus.cc:2466] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:02.233377 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a582d4f907f340179024c0377d3893e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.233611 775 raft_consensus.cc:2466] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:02.234380 1154 tablet_bootstrap.cc:492] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.234524 1154 ts_tablet_manager.cc:1397] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20250629 01:52:02.235198 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.235188 1154 raft_consensus.cc:357] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.235354 1154 raft_consensus.cc:383] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.235373 775 raft_consensus.cc:2466] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:02.235440 1154 raft_consensus.cc:738] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.235646 1154 consensus_queue.cc:260] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.236290 1154 ts_tablet_manager.cc:1428] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:02.236552 1154 tablet_bootstrap.cc:492] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.237334 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a8a9bc691e4b158276f34a51c0dc61" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.238355 1154 tablet_bootstrap.cc:654] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.246533 2791 tablet.cc:1717] T 76118a6e684448b09f244d44c909ccaa P ef5c5979bb9e475f983af7c3e9e0e019: Skipping requested alter to schema version 5, tablet already version 5
I20250629 01:52:02.248373 2765 tablet.cc:1717] T 76118a6e684448b09f244d44c909ccaa P 4712643b602e4f898752fcab6a1e5913: Skipping requested alter to schema version 5, tablet already version 5
I20250629 01:52:02.249818 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5d45854191247bf8172d0bbfd01b6db" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:02.250034 1154 tablet_bootstrap.cc:492] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.250128 1154 ts_tablet_manager.cc:1397] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20250629 01:52:02.250679 1154 raft_consensus.cc:357] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.250795 1154 raft_consensus.cc:383] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.250837 1154 raft_consensus.cc:738] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.250967 1154 consensus_queue.cc:260] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.251513 1154 ts_tablet_manager.cc:1428] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:02.251729 1154 tablet_bootstrap.cc:492] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:52:02.252257 710 consensus_peers.cc:489] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab0aa4e74d345b2a214b90b5018c82b. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.252218 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5d45854191247bf8172d0bbfd01b6db" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.253423 1154 tablet_bootstrap.cc:654] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:02.256641 850 leader_election.cc:343] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:52:02.256749 850 leader_election.cc:304] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.257112 2670 raft_consensus.cc:2747] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.257365 1154 tablet_bootstrap.cc:492] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.257459 1154 ts_tablet_manager.cc:1397] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
W20250629 01:52:02.257443 850 leader_election.cc:343] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 06e31520d3ad4c3395a59049755a4725
I20250629 01:52:02.257519 850 leader_election.cc:304] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.257824 2670 raft_consensus.cc:2747] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.258044 1154 raft_consensus.cc:357] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.258172 1154 raft_consensus.cc:383] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.258260 1154 raft_consensus.cc:738] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.258431 1154 consensus_queue.cc:260] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.259012 1154 ts_tablet_manager.cc:1428] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.259164 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd9a41fae477470281dfba4d4e552b99" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.259231 1154 tablet_bootstrap.cc:492] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.260396 1154 tablet_bootstrap.cc:654] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.264050 1156 ts_tablet_manager.cc:1918] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.264210 1156 log.cc:1199] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/303f77a729c648f9b1146d11056b242d
I20250629 01:52:02.264525 1156 ts_tablet_manager.cc:1939] T 303f77a729c648f9b1146d11056b242d P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:02.265254 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1371) took 1086 ms. Trace:
I20250629 01:52:02.268940 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 47cbc958eca14ed69fb45c3c2984aeef
I20250629 01:52:02.270613 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 2fcead51fd9b43d0a52c4669a5e23fe5 (table default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317]). Expected version 2 got 0
I20250629 01:52:02.275993 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_632fd08e723c4862bb99e1041edbd220" } new_table_name: "default.Table_e08390623cad473e9b3bc40164b9f51d" modify_external_catalogs: true
I20250629 01:52:02.276386 1154 tablet_bootstrap.cc:492] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.276473 1154 ts_tablet_manager.cc:1397] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20250629 01:52:02.277001 1154 raft_consensus.cc:357] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.277112 1154 raft_consensus.cc:383] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.277150 1154 raft_consensus.cc:738] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.277294 1154 consensus_queue.cc:260] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.277653 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet edbc80f0be2e409cb2b9f2bf206c944a
I20250629 01:52:02.277848 1154 ts_tablet_manager.cc:1428] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:02.278045 1154 tablet_bootstrap.cc:492] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.278240 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_70c13f7893b2471387ac7400f4f69c0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:02.279199 1154 tablet_bootstrap.cc:654] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.279341 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_efd88358bbde43528c5d3013bb7aaaf2" } new_table_name: "default.Table_1bcb2d8c2d424b15834889cd98119d72" modify_external_catalogs: true
W20250629 01:52:02.286744 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70c13f7893b2471387ac7400f4f69c0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:02.287815 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64ed16be3e684e32b6eb80a6a23c86ac consensus state has no leader
I20250629 01:52:02.288048 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_05a1132bda094159a72bb34e6b003609"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.288540 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05a1132bda094159a72bb34e6b003609 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.296947 884 tablet_service.cc:1515] Processing DeleteTablet for tablet edbc80f0be2e409cb2b9f2bf206c944a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:02.297709 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet edbc80f0be2e409cb2b9f2bf206c944a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edbc80f0be2e409cb2b9f2bf206c944a already in progress: creating tablet
I20250629 01:52:02.298444 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 47cbc958eca14ed69fb45c3c2984aeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.298679 2526 tablet_replica.cc:331] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:02.298898 2526 raft_consensus.cc:2241] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.299078 2526 raft_consensus.cc:2270] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.299667 2526 ts_tablet_manager.cc:1905] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.302781 1159 tablet_bootstrap.cc:492] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.302879 1159 ts_tablet_manager.cc:1397] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.098s user 0.014s sys 0.011s
I20250629 01:52:02.303637 1159 raft_consensus.cc:357] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.303869 1159 raft_consensus.cc:383] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.303963 1159 raft_consensus.cc:738] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.304263 1159 consensus_queue.cc:260] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.304910 1159 ts_tablet_manager.cc:1428] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:02.305200 1159 tablet_bootstrap.cc:492] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.306888 1159 tablet_bootstrap.cc:654] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.307348 2526 ts_tablet_manager.cc:1918] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:02.307471 2526 log.cc:1199] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/47cbc958eca14ed69fb45c3c2984aeef
I20250629 01:52:02.308077 2526 ts_tablet_manager.cc:1939] T 47cbc958eca14ed69fb45c3c2984aeef P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
W20250629 01:52:02.310037 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 44243.1us
I20250629 01:52:02.312016 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet edbc80f0be2e409cb2b9f2bf206c944a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.317801 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]) successfully deleted
I20250629 01:52:02.323449 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 47cbc958eca14ed69fb45c3c2984aeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.326144 986 leader_election.cc:304] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:02.326418 2764 raft_consensus.cc:2802] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:02.326558 2764 raft_consensus.cc:695] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:02.326802 2764 consensus_queue.cc:237] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.328573 1158 tablet_replica.cc:331] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.328743 1158 raft_consensus.cc:2241] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.328884 1158 raft_consensus.cc:2270] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:02.329070 986 leader_election.cc:343] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.329206 986 leader_election.cc:304] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.329432 1158 ts_tablet_manager.cc:1905] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.334978 1158 ts_tablet_manager.cc:1918] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:02.335106 1158 log.cc:1199] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/47cbc958eca14ed69fb45c3c2984aeef
I20250629 01:52:02.335428 1158 ts_tablet_manager.cc:1939] T 47cbc958eca14ed69fb45c3c2984aeef P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.336683 1159 tablet_bootstrap.cc:492] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.336789 1159 ts_tablet_manager.cc:1397] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.001s
I20250629 01:52:02.337335 1159 raft_consensus.cc:357] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.337456 1159 raft_consensus.cc:383] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.337495 1159 raft_consensus.cc:738] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.337657 1159 consensus_queue.cc:260] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.338057 1159 ts_tablet_manager.cc:1428] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:02.338264 1159 tablet_bootstrap.cc:492] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:02.339735 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu pid 1851: Timed out: CheckLeaks RPC to 127.0.104.61:35873 timed out after 1.000s (SENT)
I20250629 01:52:02.339864 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu with pid 1851
I20250629 01:52:02.340415 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]) successfully deleted
W20250629 01:52:02.340696 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet a1da6e7dee904f23886f6d9d75c8e49f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6e7dee904f23886f6d9d75c8e49f already in progress: creating tablet
W20250629 01:52:02.340857 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet edbc80f0be2e409cb2b9f2bf206c944a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edbc80f0be2e409cb2b9f2bf206c944a already in progress: creating tablet
I20250629 01:52:02.340991 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 47cbc958eca14ed69fb45c3c2984aeef (table default.Table_27543398d60941ce8b16591b00423498 [id=0ae8d45bb2a84f9588953b68b329f929]) successfully deleted
W20250629 01:52:02.341423 986 raft_consensus.cc:2684] T ec0ca04e5f6b42df8567deb53a3e5f53 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:52:02.342341 986 leader_election.cc:304] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:02.342545 986 raft_consensus.cc:2684] T 476a6cf28db04f67bc23ddfc5cacad87 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250629 01:52:02.349086 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03e4faf922e54c7a8bbe42694e0b899a consensus state has no leader
I20250629 01:52:02.351898 1159 tablet_bootstrap.cc:654] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.353248 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_222ce79cd46a42eea3f0354709d057ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.353765 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_222ce79cd46a42eea3f0354709d057ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.353803 886 tablet_service.cc:1468] Processing CreateTablet for tablet 490158e27490403bb8488475f3478713 (DEFAULT_TABLE table=default.Table_69c6504c160847438c542810417992c2 [id=02a7777c98fc410f85679a35541cd60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.353933 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9e2268079e3048f7b43a7ef8a95cf391 (DEFAULT_TABLE table=default.Table_0c6f823d48f243c3809066323ee8deae [id=79fa940287264ae0be41fc96a70721d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.354102 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490158e27490403bb8488475f3478713. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.354367 888 tablet_service.cc:1468] Processing CreateTablet for tablet 56ebb6d9f6284e5ea56528f1a06950b7 (DEFAULT_TABLE table=default.Table_50ba0d7936d04f23b5eced9bd61474b8 [id=bc09732e56164446a706c370e76a58fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.354645 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ebb6d9f6284e5ea56528f1a06950b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:02.355506 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad1ba1fec9bf49a5a44a193802edde0a consensus state has no leader
I20250629 01:52:02.355762 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f3b844a43dbc4e268724542a0d465f0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.356171 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3b844a43dbc4e268724542a0d465f0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.359373 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2268079e3048f7b43a7ef8a95cf391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.359853 1159 tablet_bootstrap.cc:492] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.359936 1159 ts_tablet_manager.cc:1397] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250629 01:52:02.360520 1159 raft_consensus.cc:357] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.360671 1159 raft_consensus.cc:383] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.360710 1159 raft_consensus.cc:738] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.360875 1159 consensus_queue.cc:260] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.363520 2758 raft_consensus.cc:491] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.363641 2758 raft_consensus.cc:513] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.363860 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 412a3d30c85442cd943a163cf5ea61b5 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:02.364161 2758 leader_election.cc:290] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.364804 1159 ts_tablet_manager.cc:1428] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250629 01:52:02.365386 1159 tablet_bootstrap.cc:492] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.366106 886 tablet_service.cc:1468] Processing CreateTablet for tablet 443a07b4d3994789801bb73e6e451b3b (DEFAULT_TABLE table=default.Table_c64d8f4e6caa4ef4a4feb26802b0fbd8 [id=c713c2b6614f4500a6b08c13c77c8d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.366799 1159 tablet_bootstrap.cc:654] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.366824 878 tablet_service.cc:1468] Processing CreateTablet for tablet 82173e4af9cc44d1ae25f0650ba705c0 (DEFAULT_TABLE table=default.Table_75ece33d5b574d8b83f95340d391681f [id=34c1c79f4d8e4a91b22d4ea9a4c23ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.368320 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82173e4af9cc44d1ae25f0650ba705c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.369414 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:02.370460 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:02.371054 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:02.371536 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_5794ea9989b54ab9bd7ecca30428c971 [id=dcc07a46945a4add8e667bfea51dfed7] alter complete (version 5)
I20250629 01:52:02.371938 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a65d27fa1e0a4e18b5133bb7a8ccd201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.372608 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a65d27fa1e0a4e18b5133bb7a8ccd201 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.374119 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443a07b4d3994789801bb73e6e451b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.375929 446 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:52:02.376551 446 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet cd9a41fae477470281dfba4d4e552b99 (table default.Table_e7206ec77eb44e729b77a7bda6dea373 [id=4525666163d64ba981dfff1c095c6ee2]). Expected version 2 got 0
I20250629 01:52:02.376739 446 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:02.377843 1159 tablet_bootstrap.cc:492] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
W20250629 01:52:02.377959 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 443a07b4d3994789801bb73e6e451b3b consensus state has no leader
I20250629 01:52:02.377995 1159 ts_tablet_manager.cc:1397] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20250629 01:52:02.378640 1159 raft_consensus.cc:357] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.378823 1159 raft_consensus.cc:383] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.378895 1159 raft_consensus.cc:738] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.379094 1159 consensus_queue.cc:260] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.380651 888 tablet_service.cc:1468] Processing CreateTablet for tablet d226cb8fb0134f5faebc753dca04da2d (DEFAULT_TABLE table=default.Table_f802e382de6348e9b94cf6982926fa33 [id=93994f7e593f49219177186e8bdeeade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.380946 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d226cb8fb0134f5faebc753dca04da2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.381206 1159 ts_tablet_manager.cc:1428] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:52:02.382788 1159 tablet_bootstrap.cc:492] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.383133 884 tablet_service.cc:1468] Processing CreateTablet for tablet f97db8f4217d4da9bfe62b18b850e507 (DEFAULT_TABLE table=default.Table_9f57ddf2fcf445128766cfb7fcc80f3e [id=d8c1a72737e443938fb5651ff2a606a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.383729 892 tablet_service.cc:1468] Processing CreateTablet for tablet e0d3a91c34e74637be3d72d130a8dda8 (DEFAULT_TABLE table=default.Table_5d389e95bd2c44a28cbcc84d1d47a30f [id=0a9f6b39281c4ce3b95380b9222af2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.383733 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97db8f4217d4da9bfe62b18b850e507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.384155 1159 tablet_bootstrap.cc:654] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.384295 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d3a91c34e74637be3d72d130a8dda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.388897 879 tablet_service.cc:1468] Processing CreateTablet for tablet f7105eec690746daa83266991890058b (DEFAULT_TABLE table=default.Table_76b76a14142749efb0305e749805438f [id=dc694591bccc4aae9145b019d7f2fe3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.389453 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7105eec690746daa83266991890058b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.393301 886 tablet_service.cc:1468] Processing CreateTablet for tablet 77db4198f8384e6aa98f47e075f62a37 (DEFAULT_TABLE table=default.Table_8147f0891d8a40e185acaad58f89fc75 [id=0368c6255a4d438c8e04238f13087242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.393344 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef3060d46ac24330be49103920dc1dee" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.393606 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77db4198f8384e6aa98f47e075f62a37. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:02.393921 988 leader_election.cc:343] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.399199 878 tablet_service.cc:1468] Processing CreateTablet for tablet c2e36e81d5d14fecb5bed176c6fc996d (DEFAULT_TABLE table=default.Table_3610a48595354365bc24c326de4fc485 [id=961f23a5aeff4061a034c708547db086]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.399565 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2e36e81d5d14fecb5bed176c6fc996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.400725 1159 tablet_bootstrap.cc:492] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.400863 1159 ts_tablet_manager.cc:1397] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20250629 01:52:02.401418 1159 raft_consensus.cc:357] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.401584 1159 raft_consensus.cc:383] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.401670 1159 raft_consensus.cc:738] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.401862 1159 consensus_queue.cc:260] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.404008 1159 ts_tablet_manager.cc:1428] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:52:02.405645 1159 tablet_bootstrap.cc:492] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:02.406010 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 64693.8us
I20250629 01:52:02.407002 1159 tablet_bootstrap.cc:654] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.406937 1031 tablet_service.cc:1468] Processing CreateTablet for tablet f7105eec690746daa83266991890058b (DEFAULT_TABLE table=default.Table_76b76a14142749efb0305e749805438f [id=dc694591bccc4aae9145b019d7f2fe3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.407552 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7105eec690746daa83266991890058b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.413110 877 tablet_service.cc:1468] Processing CreateTablet for tablet e6150a9f828246eab235eba80f0966f7 (DEFAULT_TABLE table=default.Table_12dfc35bb7ba4e1792cce572411cdeb6 [id=c60b994908654e618614fc5bf18c8801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.413360 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6150a9f828246eab235eba80f0966f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.415442 881 tablet_service.cc:1468] Processing CreateTablet for tablet b5712d48e7ce49eea600bbc62620c46c (DEFAULT_TABLE table=default.Table_2ed6d41c999f40d994a3609923eeffd3 [id=2a82c723bd344f539d3a5a439a9c6572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.415688 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5712d48e7ce49eea600bbc62620c46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.415880 883 tablet_service.cc:1468] Processing CreateTablet for tablet 56e7817befe8410bbb2fbe25d9f5b337 (DEFAULT_TABLE table=default.Table_2d84c16e159a46469c09eba21ca1f592 [id=6a0b922e3e974c0aa578ee085ca570a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.416147 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e7817befe8410bbb2fbe25d9f5b337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.416420 885 tablet_service.cc:1468] Processing CreateTablet for tablet 6298871f47a94ff58c9931e0fa913771 (DEFAULT_TABLE table=default.Table_19d55c4c3f444c41b10920604792dccb [id=6a79e65c9b304171b1fdeba3ba898c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.416678 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6298871f47a94ff58c9931e0fa913771. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.417007 895 tablet_service.cc:1468] Processing CreateTablet for tablet e69bc16334a84d46a21e02c307be401f (DEFAULT_TABLE table=default.Table_b80c833964504cee9ae5d15210febf5f [id=bd5bdfae0b5740ac8e174e19a38c1300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.417243 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69bc16334a84d46a21e02c307be401f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.422520 878 tablet_service.cc:1468] Processing CreateTablet for tablet 1b5ac7a83fac4b168a4d3ea6ed301898 (DEFAULT_TABLE table=default.Table_acd11ca3b08f4e69a5df56589bf21a9d [id=86708bf49b5b47008df03a3b6c6fc9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.422801 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5ac7a83fac4b168a4d3ea6ed301898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.423671 1024 tablet_service.cc:1468] Processing CreateTablet for tablet f97db8f4217d4da9bfe62b18b850e507 (DEFAULT_TABLE table=default.Table_9f57ddf2fcf445128766cfb7fcc80f3e [id=d8c1a72737e443938fb5651ff2a606a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.423959 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97db8f4217d4da9bfe62b18b850e507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.424214 986 leader_election.cc:304] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:02.424551 986 leader_election.cc:304] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:02.424737 2760 raft_consensus.cc:2802] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.424789 2760 raft_consensus.cc:491] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.424834 2760 raft_consensus.cc:3058] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.426375 2758 raft_consensus.cc:2802] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.426435 2758 raft_consensus.cc:491] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.426478 2758 raft_consensus.cc:3058] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.427017 2760 raft_consensus.cc:513] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.427474 2760 leader_election.cc:290] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.428345 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a582d4f907f340179024c0377d3893e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:02.428766 2758 raft_consensus.cc:513] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.428848 988 leader_election.cc:343] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.428395 886 tablet_service.cc:1468] Processing CreateTablet for tablet 5db79f2d97ec4c87931522aa4c27965f (DEFAULT_TABLE table=default.Table_5bbd9c09a3a2490ab18ec643b60078b1 [id=3a63d6e3449a4310977ab3f90f38e182]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.429235 2758 leader_election.cc:290] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.429486 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db79f2d97ec4c87931522aa4c27965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.433950 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:02.434763 988 leader_election.cc:343] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:02.436687 988 connection.cc:537] client connection to 127.0.104.61:35873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250629 01:52:02.440210 442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.61:35873 (error 108) [suppressed 5 similar messages]
I20250629 01:52:02.444577 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 9e2268079e3048f7b43a7ef8a95cf391 (DEFAULT_TABLE table=default.Table_0c6f823d48f243c3809066323ee8deae [id=79fa940287264ae0be41fc96a70721d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.444964 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2268079e3048f7b43a7ef8a95cf391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.445760 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 56ebb6d9f6284e5ea56528f1a06950b7 (DEFAULT_TABLE table=default.Table_50ba0d7936d04f23b5eced9bd61474b8 [id=bc09732e56164446a706c370e76a58fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.444680 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 490158e27490403bb8488475f3478713 (DEFAULT_TABLE table=default.Table_69c6504c160847438c542810417992c2 [id=02a7777c98fc410f85679a35541cd60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.446136 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ebb6d9f6284e5ea56528f1a06950b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.446841 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 443a07b4d3994789801bb73e6e451b3b (DEFAULT_TABLE table=default.Table_c64d8f4e6caa4ef4a4feb26802b0fbd8 [id=c713c2b6614f4500a6b08c13c77c8d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.447077 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490158e27490403bb8488475f3478713. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.447265 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443a07b4d3994789801bb73e6e451b3b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:02.460364 986 leader_election.cc:343] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.460382 2670 raft_consensus.cc:491] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.460525 2670 raft_consensus.cc:513] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.460510 986 leader_election.cc:304] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.461122 2670 leader_election.cc:290] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:02.461197 2758 raft_consensus.cc:2747] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.461897 1159 tablet_bootstrap.cc:492] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.461995 1159 ts_tablet_manager.cc:1397] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.003s
I20250629 01:52:02.462594 1159 raft_consensus.cc:357] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.462671 2758 raft_consensus.cc:491] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.462723 1159 raft_consensus.cc:383] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.462770 1159 raft_consensus.cc:738] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.462822 2758 raft_consensus.cc:513] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.462921 1159 consensus_queue.cc:260] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.463299 1159 ts_tablet_manager.cc:1428] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:02.463405 2758 leader_election.cc:290] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.463521 1159 tablet_bootstrap.cc:492] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.467317 1159 tablet_bootstrap.cc:654] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.467835 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43b845c3e7174618bd5c394d7206a634" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:02.468393 988 leader_election.cc:343] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.468712 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 82173e4af9cc44d1ae25f0650ba705c0 (DEFAULT_TABLE table=default.Table_75ece33d5b574d8b83f95340d391681f [id=34c1c79f4d8e4a91b22d4ea9a4c23ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.469007 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82173e4af9cc44d1ae25f0650ba705c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.469403 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d226cb8fb0134f5faebc753dca04da2d (DEFAULT_TABLE table=default.Table_f802e382de6348e9b94cf6982926fa33 [id=93994f7e593f49219177186e8bdeeade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.469645 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d226cb8fb0134f5faebc753dca04da2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.472774 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e0d3a91c34e74637be3d72d130a8dda8 (DEFAULT_TABLE table=default.Table_5d389e95bd2c44a28cbcc84d1d47a30f [id=0a9f6b39281c4ce3b95380b9222af2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.473079 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d3a91c34e74637be3d72d130a8dda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.473508 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 77db4198f8384e6aa98f47e075f62a37 (DEFAULT_TABLE table=default.Table_8147f0891d8a40e185acaad58f89fc75 [id=0368c6255a4d438c8e04238f13087242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.474041 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77db4198f8384e6aa98f47e075f62a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.474392 1019 tablet_service.cc:1468] Processing CreateTablet for tablet c2e36e81d5d14fecb5bed176c6fc996d (DEFAULT_TABLE table=default.Table_3610a48595354365bc24c326de4fc485 [id=961f23a5aeff4061a034c708547db086]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.474723 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2e36e81d5d14fecb5bed176c6fc996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.309422 1155 tablet_bootstrap.cc:654] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.311606 747 tablet_service.cc:1515] Processing DeleteTablet for tablet edbc80f0be2e409cb2b9f2bf206c944a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.323290 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 47cbc958eca14ed69fb45c3c2984aeef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.328388 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd9a41fae477470281dfba4d4e552b99" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.423127 742 tablet_service.cc:1468] Processing CreateTablet for tablet f7105eec690746daa83266991890058b (DEFAULT_TABLE table=default.Table_76b76a14142749efb0305e749805438f [id=dc694591bccc4aae9145b019d7f2fe3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.478039 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7105eec690746daa83266991890058b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.423969 745 tablet_service.cc:1468] Processing CreateTablet for tablet f97db8f4217d4da9bfe62b18b850e507 (DEFAULT_TABLE table=default.Table_9f57ddf2fcf445128766cfb7fcc80f3e [id=d8c1a72737e443938fb5651ff2a606a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.481742 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f97db8f4217d4da9bfe62b18b850e507. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:02.489358 710 consensus_peers.cc:489] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecbfc4e45dc9435bbd7a55ba7b7e4155. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:02.493299 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35873, user_credentials={real_user=slave}} blocked reactor thread for 53088.5us
W20250629 01:52:02.328583 712 leader_election.cc:343] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.444172 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9e2268079e3048f7b43a7ef8a95cf391 (DEFAULT_TABLE table=default.Table_0c6f823d48f243c3809066323ee8deae [id=79fa940287264ae0be41fc96a70721d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.445163 756 tablet_service.cc:1468] Processing CreateTablet for tablet 490158e27490403bb8488475f3478713 (DEFAULT_TABLE table=default.Table_69c6504c160847438c542810417992c2 [id=02a7777c98fc410f85679a35541cd60e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.495005 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490158e27490403bb8488475f3478713. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.446373 751 tablet_service.cc:1468] Processing CreateTablet for tablet 56ebb6d9f6284e5ea56528f1a06950b7 (DEFAULT_TABLE table=default.Table_50ba0d7936d04f23b5eced9bd61474b8 [id=bc09732e56164446a706c370e76a58fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.500414 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ebb6d9f6284e5ea56528f1a06950b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.452340 744 tablet_service.cc:1468] Processing CreateTablet for tablet 443a07b4d3994789801bb73e6e451b3b (DEFAULT_TABLE table=default.Table_c64d8f4e6caa4ef4a4feb26802b0fbd8 [id=c713c2b6614f4500a6b08c13c77c8d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.503559 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443a07b4d3994789801bb73e6e451b3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.460651 2548 raft_consensus.cc:491] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.508178 2548 raft_consensus.cc:513] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.508842 2548 leader_election.cc:290] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.468904 748 tablet_service.cc:1468] Processing CreateTablet for tablet 82173e4af9cc44d1ae25f0650ba705c0 (DEFAULT_TABLE table=default.Table_75ece33d5b574d8b83f95340d391681f [id=34c1c79f4d8e4a91b22d4ea9a4c23ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.509261 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82173e4af9cc44d1ae25f0650ba705c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.472605 740 tablet_service.cc:1468] Processing CreateTablet for tablet d226cb8fb0134f5faebc753dca04da2d (DEFAULT_TABLE table=default.Table_f802e382de6348e9b94cf6982926fa33 [id=93994f7e593f49219177186e8bdeeade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.472997 757 tablet_service.cc:1468] Processing CreateTablet for tablet e0d3a91c34e74637be3d72d130a8dda8 (DEFAULT_TABLE table=default.Table_5d389e95bd2c44a28cbcc84d1d47a30f [id=0a9f6b39281c4ce3b95380b9222af2ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.473742 743 tablet_service.cc:1468] Processing CreateTablet for tablet 77db4198f8384e6aa98f47e075f62a37 (DEFAULT_TABLE table=default.Table_8147f0891d8a40e185acaad58f89fc75 [id=0368c6255a4d438c8e04238f13087242]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.473938 750 tablet_service.cc:1468] Processing CreateTablet for tablet c2e36e81d5d14fecb5bed176c6fc996d (DEFAULT_TABLE table=default.Table_3610a48595354365bc24c326de4fc485 [id=961f23a5aeff4061a034c708547db086]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.512483 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2e36e81d5d14fecb5bed176c6fc996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.516208 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0d3a91c34e74637be3d72d130a8dda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.516682 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77db4198f8384e6aa98f47e075f62a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.521416 2670 raft_consensus.cc:491] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.521529 2670 raft_consensus.cc:513] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.522020 2670 leader_election.cc:290] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:02.523411 2758 raft_consensus.cc:491] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.523581 2758 raft_consensus.cc:513] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.523813 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873): Couldn't send request to peer 0f4258c004e04bcf8216bb7c8fd56c54. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.524267 2758 leader_election.cc:290] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:02.529592 710 leader_election.cc:343] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: dc841eb171d04b118b1e705ceed4c861
I20250629 01:52:02.539072 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d226cb8fb0134f5faebc753dca04da2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.556810 1159 tablet_bootstrap.cc:492] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.556950 1159 ts_tablet_manager.cc:1397] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.093s user 0.022s sys 0.017s
I20250629 01:52:02.557766 1159 raft_consensus.cc:357] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.557943 1159 raft_consensus.cc:383] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.558002 1159 raft_consensus.cc:738] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.558192 1159 consensus_queue.cc:260] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.558701 1159 ts_tablet_manager.cc:1428] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:02.559674 2670 raft_consensus.cc:491] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.559845 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e2268079e3048f7b43a7ef8a95cf391. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.559795 2670 raft_consensus.cc:513] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.560359 2670 leader_election.cc:290] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:02.566910 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.574592 712 leader_election.cc:304] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.574908 2548 raft_consensus.cc:2747] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:02.575255 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 246632us
W20250629 01:52:02.575531 712 leader_election.cc:343] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:02.576335 712 leader_election.cc:343] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.580703 916 raft_consensus.cc:3058] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.586002 1155 tablet_bootstrap.cc:492] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.586149 1155 ts_tablet_manager.cc:1397] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.436s user 0.003s sys 0.000s
I20250629 01:52:02.586755 1155 raft_consensus.cc:357] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.586961 1155 raft_consensus.cc:383] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250629 01:52:02.586987 988 leader_election.cc:343] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.587051 1155 raft_consensus.cc:738] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.587239 1155 consensus_queue.cc:260] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.592463 1155 ts_tablet_manager.cc:1428] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250629 01:52:02.592701 1155 tablet_bootstrap.cc:492] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.594034 1155 tablet_bootstrap.cc:654] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.596467 1156 rpcz_store.cc:276] 0629 01:52:01.178840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.178882 (+ 42us) service_pool.cc:225] Handling call
0629 01:52:02.089659 (+910777us) ts_tablet_manager.cc:1145] Deleting tablet 303f77a729c648f9b1146d11056b242d
0629 01:52:02.089666 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:02.173521 (+ 83855us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.263969 (+ 90448us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.265195 (+ 1226us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":910647}
I20250629 01:52:02.597222 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1373) took 1417 ms. Trace:
I20250629 01:52:02.597371 1156 rpcz_store.cc:276] 0629 01:52:01.179634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.179683 (+ 49us) service_pool.cc:225] Handling call
0629 01:52:02.597064 (+1417381us) ts_tablet_manager.cc:1145] Deleting tablet ce90c045115844b0b96143f91f1be1d4
0629 01:52:02.597075 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:02.597204 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1417243}
I20250629 01:52:02.598013 1156 tablet_replica.cc:331] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:02.598287 1156 raft_consensus.cc:2241] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.598654 1156 raft_consensus.cc:2270] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.599243 1156 ts_tablet_manager.cc:1905] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.603593 850 leader_election.cc:304] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.603971 2670 raft_consensus.cc:2747] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.605058 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc841eb171d04b118b1e705ceed4c861" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.605803 2764 raft_consensus.cc:491] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.608584 1155 tablet_bootstrap.cc:492] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.608736 1155 ts_tablet_manager.cc:1397] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20250629 01:52:02.611959 2534 raft_consensus.cc:491] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.613615 2534 raft_consensus.cc:513] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.614032 2534 leader_election.cc:290] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.614923 1155 raft_consensus.cc:357] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.615095 1155 raft_consensus.cc:383] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.615187 1155 raft_consensus.cc:738] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.615454 1155 consensus_queue.cc:260] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.616245 1155 ts_tablet_manager.cc:1428] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:02.616633 1155 tablet_bootstrap.cc:492] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.618181 1155 tablet_bootstrap.cc:654] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.620824 2670 raft_consensus.cc:491] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.620963 2670 raft_consensus.cc:513] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.621616 2670 leader_election.cc:290] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:02.625142 2670 raft_consensus.cc:491] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.625276 2670 raft_consensus.cc:513] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.625845 2670 leader_election.cc:290] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:02.627117 1155 tablet_bootstrap.cc:492] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.627262 1155 ts_tablet_manager.cc:1397] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250629 01:52:02.627995 1155 raft_consensus.cc:357] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.628206 1155 raft_consensus.cc:383] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.628288 1155 raft_consensus.cc:738] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.628492 1155 consensus_queue.cc:260] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.628873 1155 ts_tablet_manager.cc:1428] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.629093 1155 tablet_bootstrap.cc:492] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.630259 1155 tablet_bootstrap.cc:654] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.626943 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.632768 744 tablet_service.cc:1468] Processing CreateTablet for tablet e6150a9f828246eab235eba80f0966f7 (DEFAULT_TABLE table=default.Table_12dfc35bb7ba4e1792cce572411cdeb6 [id=c60b994908654e618614fc5bf18c8801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.633214 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6150a9f828246eab235eba80f0966f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.641283 753 tablet_service.cc:1468] Processing CreateTablet for tablet 5db79f2d97ec4c87931522aa4c27965f (DEFAULT_TABLE table=default.Table_5bbd9c09a3a2490ab18ec643b60078b1 [id=3a63d6e3449a4310977ab3f90f38e182]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.641711 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db79f2d97ec4c87931522aa4c27965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.645514 1052 raft_consensus.cc:2466] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:52:02.645561 986 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.60:32789, user_credentials={real_user=slave}} blocked reactor thread for 167063us
I20250629 01:52:02.645776 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 5db79f2d97ec4c87931522aa4c27965f (DEFAULT_TABLE table=default.Table_5bbd9c09a3a2490ab18ec643b60078b1 [id=3a63d6e3449a4310977ab3f90f38e182]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.645885 710 leader_election.cc:304] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.646100 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db79f2d97ec4c87931522aa4c27965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.646082 2534 raft_consensus.cc:2802] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.646137 2534 raft_consensus.cc:491] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.646185 2534 raft_consensus.cc:3058] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.647383 1155 tablet_bootstrap.cc:492] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.647468 1155 ts_tablet_manager.cc:1397] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20250629 01:52:02.647993 1155 raft_consensus.cc:357] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.648131 1155 raft_consensus.cc:383] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.648172 1155 raft_consensus.cc:738] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.648303 1155 consensus_queue.cc:260] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.648351 2534 raft_consensus.cc:513] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.648689 1155 ts_tablet_manager.cc:1428] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:02.648808 2534 leader_election.cc:290] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.648874 1155 tablet_bootstrap.cc:492] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.649148 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.649451 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d87639255ec46d98fcfb0f9638ee39a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.650024 1155 tablet_bootstrap.cc:654] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.650182 2764 raft_consensus.cc:513] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.651016 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e6150a9f828246eab235eba80f0966f7 (DEFAULT_TABLE table=default.Table_12dfc35bb7ba4e1792cce572411cdeb6 [id=c60b994908654e618614fc5bf18c8801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.651026 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 1b5ac7a83fac4b168a4d3ea6ed301898 (DEFAULT_TABLE table=default.Table_acd11ca3b08f4e69a5df56589bf21a9d [id=86708bf49b5b47008df03a3b6c6fc9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.651453 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6150a9f828246eab235eba80f0966f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.652290 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d0fbb691514d51a1ca71f32a666c2d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:02.656386 850 leader_election.cc:343] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8975c1deaaf44761be791f460b53ec74
W20250629 01:52:02.656526 850 leader_election.cc:343] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 5b8a2d1a7f7146f4968590914374ba23
W20250629 01:52:02.656703 850 leader_election.cc:343] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f3bdc8bda8a940be865679a05b90bd08
W20250629 01:52:02.657199 988 leader_election.cc:343] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.657240 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:02.657214 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5ac7a83fac4b168a4d3ea6ed301898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.657471 2764 leader_election.cc:290] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.658092 1051 raft_consensus.cc:3058] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.659317 742 tablet_service.cc:1468] Processing CreateTablet for tablet 1b5ac7a83fac4b168a4d3ea6ed301898 (DEFAULT_TABLE table=default.Table_acd11ca3b08f4e69a5df56589bf21a9d [id=86708bf49b5b47008df03a3b6c6fc9bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.659584 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5ac7a83fac4b168a4d3ea6ed301898. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.660382 1154 tablet_bootstrap.cc:492] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.660892 1154 ts_tablet_manager.cc:1397] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.383s user 0.113s sys 0.153s
I20250629 01:52:02.661525 1154 raft_consensus.cc:357] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.661963 1154 raft_consensus.cc:383] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.662092 1154 raft_consensus.cc:738] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.663637 1154 consensus_queue.cc:260] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.664278 1154 ts_tablet_manager.cc:1428] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250629 01:52:02.664618 2794 tablet.cc:1722] T e1e22a0c02054c7db514eefe20ee3311 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:02.666069 1154 tablet_bootstrap.cc:492] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.669011 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef3060d46ac24330be49103920dc1dee" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.669288 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a582d4f907f340179024c0377d3893e2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:02.669406 775 raft_consensus.cc:3058] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.669680 1159 tablet_bootstrap.cc:492] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.670827 1159 tablet_bootstrap.cc:654] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.670956 1051 raft_consensus.cc:2466] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:02.671216 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 56e7817befe8410bbb2fbe25d9f5b337 (DEFAULT_TABLE table=default.Table_2d84c16e159a46469c09eba21ca1f592 [id=6a0b922e3e974c0aa578ee085ca570a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.671679 775 raft_consensus.cc:2466] T a582d4f907f340179024c0377d3893e2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:02.671917 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:02.671958 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e7817befe8410bbb2fbe25d9f5b337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.672036 776 raft_consensus.cc:3058] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.672600 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d0fbb691514d51a1ca71f32a666c2d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.672720 774 raft_consensus.cc:2466] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:02.673382 1154 tablet_bootstrap.cc:654] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.674000 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c6239dd09a343d59da51d114f48658a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.674232 776 raft_consensus.cc:2466] T 4030891de41444e0b387492e570a11a5 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:02.674441 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.674574 771 raft_consensus.cc:2391] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:52:02.675336 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 6298871f47a94ff58c9931e0fa913771 (DEFAULT_TABLE table=default.Table_19d55c4c3f444c41b10920604792dccb [id=6a79e65c9b304171b1fdeba3ba898c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.675388 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b5712d48e7ce49eea600bbc62620c46c (DEFAULT_TABLE table=default.Table_2ed6d41c999f40d994a3609923eeffd3 [id=2a82c723bd344f539d3a5a439a9c6572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.675716 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5712d48e7ce49eea600bbc62620c46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.675920 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6298871f47a94ff58c9931e0fa913771. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.678970 710 leader_election.cc:304] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.679832 2548 raft_consensus.cc:2802] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:02.679929 2548 raft_consensus.cc:695] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:02.680110 2548 consensus_queue.cc:237] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.681171 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43b845c3e7174618bd5c394d7206a634" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.681284 777 raft_consensus.cc:2466] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:02.682103 750 tablet_service.cc:1468] Processing CreateTablet for tablet e69bc16334a84d46a21e02c307be401f (DEFAULT_TABLE table=default.Table_b80c833964504cee9ae5d15210febf5f [id=bd5bdfae0b5740ac8e174e19a38c1300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.682535 751 tablet_service.cc:1468] Processing CreateTablet for tablet b5712d48e7ce49eea600bbc62620c46c (DEFAULT_TABLE table=default.Table_2ed6d41c999f40d994a3609923eeffd3 [id=2a82c723bd344f539d3a5a439a9c6572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.682869 757 tablet_service.cc:1468] Processing CreateTablet for tablet 6298871f47a94ff58c9931e0fa913771 (DEFAULT_TABLE table=default.Table_19d55c4c3f444c41b10920604792dccb [id=6a79e65c9b304171b1fdeba3ba898c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.684296 1031 tablet_service.cc:1468] Processing CreateTablet for tablet e69bc16334a84d46a21e02c307be401f (DEFAULT_TABLE table=default.Table_b80c833964504cee9ae5d15210febf5f [id=bd5bdfae0b5740ac8e174e19a38c1300]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.684558 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69bc16334a84d46a21e02c307be401f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.686983 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d87639255ec46d98fcfb0f9638ee39a" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:02.688694 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 303f77a729c648f9b1146d11056b242d (table default.Table_fcb861de87ad468584f2f0c88b9db3b6 [id=7c4d6f0956f142d195bd44428072be8f]) successfully deleted
W20250629 01:52:02.688902 440 catalog_manager.cc:4895] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ce90c045115844b0b96143f91f1be1d4 because the tablet was not found. No further retry: Not found: Tablet not found: ce90c045115844b0b96143f91f1be1d4
I20250629 01:52:02.691485 1159 tablet_bootstrap.cc:492] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.691646 1159 ts_tablet_manager.cc:1397] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.004s
I20250629 01:52:02.692276 1159 raft_consensus.cc:357] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.692449 1159 raft_consensus.cc:383] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.692520 1159 raft_consensus.cc:738] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.692704 1159 consensus_queue.cc:260] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.693133 1159 ts_tablet_manager.cc:1428] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:02.693365 1159 tablet_bootstrap.cc:492] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.694432 1159 tablet_bootstrap.cc:654] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.697875 2534 consensus_queue.cc:1035] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:02.698593 2760 raft_consensus.cc:491] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.698697 2760 raft_consensus.cc:513] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.699105 2760 leader_election.cc:290] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.699925 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8635903caac9480184c32e6689870485" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:02.700366 988 leader_election.cc:343] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.704789 745 tablet_service.cc:1468] Processing CreateTablet for tablet 56e7817befe8410bbb2fbe25d9f5b337 (DEFAULT_TABLE table=default.Table_2d84c16e159a46469c09eba21ca1f592 [id=6a0b922e3e974c0aa578ee085ca570a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.708680 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69bc16334a84d46a21e02c307be401f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.712330 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5712d48e7ce49eea600bbc62620c46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.716326 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6298871f47a94ff58c9931e0fa913771. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.716462 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e7817befe8410bbb2fbe25d9f5b337. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.721747 2670 raft_consensus.cc:491] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.721853 2670 raft_consensus.cc:513] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.722304 2670 leader_election.cc:290] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:02.733554 850 leader_election.cc:343] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 4d9ef77181ba48f3a6de16b945c8df1a
I20250629 01:52:02.735147 1155 tablet_bootstrap.cc:492] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.735333 1155 ts_tablet_manager.cc:1397] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.022s
I20250629 01:52:02.736204 1155 raft_consensus.cc:357] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.736423 1155 raft_consensus.cc:383] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.736513 1155 raft_consensus.cc:738] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.736784 1155 consensus_queue.cc:260] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.737406 1155 ts_tablet_manager.cc:1428] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.737705 1155 tablet_bootstrap.cc:492] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.739404 1155 tablet_bootstrap.cc:654] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:02.740435 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 51000us
I20250629 01:52:02.744026 1155 tablet_bootstrap.cc:492] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.744405 1155 ts_tablet_manager.cc:1397] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250629 01:52:02.745420 1155 raft_consensus.cc:357] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.746121 1155 raft_consensus.cc:383] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.746259 1155 raft_consensus.cc:738] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.746562 1155 consensus_queue.cc:260] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.747321 1155 ts_tablet_manager.cc:1428] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:02.747859 1155 tablet_bootstrap.cc:492] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.747802 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.754712 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:02.756049 1154 tablet_bootstrap.cc:492] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.756884 850 leader_election.cc:304] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.757038 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 72b2caf969db48458ea007ae03a0640f
I20250629 01:52:02.756354 1154 ts_tablet_manager.cc:1397] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.001s
I20250629 01:52:02.757222 850 leader_election.cc:304] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.757506 450 catalog_manager.cc:6652] ReplaceTablet: tablet 72b2caf969db48458ea007ae03a0640f deleted and replaced by tablet 28bc247520074b70973312e183ed6796
I20250629 01:52:02.757880 2670 raft_consensus.cc:2747] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.757900 1154 raft_consensus.cc:357] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.758019 1154 raft_consensus.cc:383] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.758036 2670 raft_consensus.cc:2747] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.758066 1154 raft_consensus.cc:738] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.758244 1154 consensus_queue.cc:260] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.758754 1154 ts_tablet_manager.cc:1428] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:02.759536 1154 tablet_bootstrap.cc:492] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:52:02.759855 710 consensus_peers.cc:489] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aa8dfd712c043ad9f8ff56b064bb8fd. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.761793 1154 tablet_bootstrap.cc:654] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.761945 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 72b2caf969db48458ea007ae03a0640f with delete_type TABLET_DATA_DELETED (replaced by tablet 28bc247520074b70973312e183ed6796) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:02.763526 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 72b2caf969db48458ea007ae03a0640f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72b2caf969db48458ea007ae03a0640f already in progress: creating tablet
I20250629 01:52:02.765883 1154 tablet_bootstrap.cc:492] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.765998 1154 ts_tablet_manager.cc:1397] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20250629 01:52:02.766350 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2692a0c5508d4f12b3f8f784f1f1bbd4
I20250629 01:52:02.766481 1154 raft_consensus.cc:357] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.766598 1154 raft_consensus.cc:383] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.766639 1154 raft_consensus.cc:738] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.766793 1154 consensus_queue.cc:260] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.767230 1154 ts_tablet_manager.cc:1428] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:02.768317 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9370ec79715e4e5fa4766f202765f04b
I20250629 01:52:02.772513 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 412a3d30c85442cd943a163cf5ea61b5
I20250629 01:52:02.772733 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 1ac60a024a074953b513ae4f60840d25 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:02.772843 452 catalog_manager.cc:6652] ReplaceTablet: tablet 412a3d30c85442cd943a163cf5ea61b5 deleted and replaced by tablet 08dd5ec4c05f4408a3b832161199140b
I20250629 01:52:02.774083 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_a9833281c36943b1a5a8009f5fc437bd [id=26e9b4aa4f6e4c07a700e299dd461e1e] alter complete (version 1)
I20250629 01:52:02.774108 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_589cda708d5c486cb3cf4cee7b376856 [id=13e374d69c0d467282f126d666c25317] alter complete (version 2)
W20250629 01:52:02.767541 986 leader_election.cc:343] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.768168 2760 raft_consensus.cc:491] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.779605 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 9370ec79715e4e5fa4766f202765f04b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.779827 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba5887dffc1460a89208447ecc6a198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.779681 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.779297 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 412a3d30c85442cd943a163cf5ea61b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 08dd5ec4c05f4408a3b832161199140b) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.779590 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.780479 1158 tablet_replica.cc:331] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.779502 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.780042 986 leader_election.cc:304] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.780634 1158 raft_consensus.cc:2241] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.780063 2760 raft_consensus.cc:513] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.780733 1158 raft_consensus.cc:2270] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.781000 986 leader_election.cc:304] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:02.781070 2760 leader_election.cc:290] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.781121 1158 ts_tablet_manager.cc:1905] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.781179 2764 raft_consensus.cc:2747] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.781424 2764 raft_consensus.cc:2802] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:02.781502 2764 raft_consensus.cc:695] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:02.781663 2764 consensus_queue.cc:237] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.779726 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.782976 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 72b2caf969db48458ea007ae03a0640f with delete_type TABLET_DATA_DELETED (replaced by tablet 28bc247520074b70973312e183ed6796) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.779562 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b845c3e7174618bd5c394d7206a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:02.767936 1154 tablet_bootstrap.cc:492] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.789146 1154 tablet_bootstrap.cc:654] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.790263 1158 ts_tablet_manager.cc:1918] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.790437 1158 log.cc:1199] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/3ba5887dffc1460a89208447ecc6a198
I20250629 01:52:02.791141 1158 ts_tablet_manager.cc:1939] T 3ba5887dffc1460a89208447ecc6a198 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:02.792304 1158 tablet_replica.cc:331] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:02.792717 1158 raft_consensus.cc:2241] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.792899 1158 raft_consensus.cc:2270] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.793401 1158 ts_tablet_manager.cc:1905] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.797046 1154 tablet_bootstrap.cc:492] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.797187 1154 ts_tablet_manager.cc:1397] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
W20250629 01:52:02.797250 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9370ec79715e4e5fa4766f202765f04b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9370ec79715e4e5fa4766f202765f04b already in progress: creating tablet
I20250629 01:52:02.797523 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 3ba5887dffc1460a89208447ecc6a198 (table default.Table_f3aaa9e9eb034a5cbbd31b1529d0080a [id=7663b74fe2374460b050e5b3fc56ad49]) successfully deleted
I20250629 01:52:02.797896 1154 raft_consensus.cc:357] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.798127 1154 raft_consensus.cc:383] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.798207 1154 raft_consensus.cc:738] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.798403 1154 consensus_queue.cc:260] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.799651 1154 ts_tablet_manager.cc:1428] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:02.800518 1154 tablet_bootstrap.cc:492] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.801759 1154 tablet_bootstrap.cc:654] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.803107 1156 ts_tablet_manager.cc:1918] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.803292 1156 log.cc:1199] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/3bdc6a26b3594d6fb33a27cac5b29f4f
I20250629 01:52:02.803368 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 412a3d30c85442cd943a163cf5ea61b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 08dd5ec4c05f4408a3b832161199140b) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.803491 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 9370ec79715e4e5fa4766f202765f04b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.803682 1156 ts_tablet_manager.cc:1939] T 3bdc6a26b3594d6fb33a27cac5b29f4f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:02.804819 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1372) took 1625 ms. Trace:
I20250629 01:52:02.804960 1156 rpcz_store.cc:276] 0629 01:52:01.179540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.179788 (+ 248us) service_pool.cc:225] Handling call
0629 01:52:02.597960 (+1418172us) ts_tablet_manager.cc:1145] Deleting tablet 3bdc6a26b3594d6fb33a27cac5b29f4f
0629 01:52:02.597972 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:02.747502 (+149530us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.787131 (+ 39629us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.804799 (+ 17668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418026}
I20250629 01:52:02.805790 1156 tablet_replica.cc:331] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:02.806061 1156 raft_consensus.cc:2241] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.806269 1156 raft_consensus.cc:2270] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.806901 1156 ts_tablet_manager.cc:1905] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.808485 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 3bdc6a26b3594d6fb33a27cac5b29f4f (table default.Table_99703cee14e34f2ebe6028b072a5b072 [id=2ba8a990fc3a474aac09caa7380ccb80]) successfully deleted
I20250629 01:52:02.808755 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:02.809841 1154 tablet_bootstrap.cc:492] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.809983 1154 ts_tablet_manager.cc:1397] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20250629 01:52:02.810801 1154 raft_consensus.cc:357] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.811014 1154 raft_consensus.cc:383] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.811106 1154 raft_consensus.cc:738] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.811887 1154 consensus_queue.cc:260] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.813199 1154 ts_tablet_manager.cc:1428] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:02.813683 2764 raft_consensus.cc:491] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.813772 2764 raft_consensus.cc:513] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.814167 2764 leader_election.cc:290] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.814433 1155 tablet_bootstrap.cc:654] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.814733 1154 tablet_bootstrap.cc:492] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.815682 1154 tablet_bootstrap.cc:654] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.819690 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1fbaed01d5a0409b94dc53e0e4a7cdc0" } modify_external_catalogs: true
I20250629 01:52:02.819854 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1fbaed01d5a0409b94dc53e0e4a7cdc0" } modify_external_catalogs: true
I20250629 01:52:02.819975 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_da80d6865112454b8f206b478bbda7cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.820433 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da80d6865112454b8f206b478bbda7cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.820350 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a24244a6087d4f0d98c841951fc3bb1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.820729 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a24244a6087d4f0d98c841951fc3bb1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:02.821499 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 64271.7us
W20250629 01:52:02.821699 850 leader_election.cc:343] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 8975c1deaaf44761be791f460b53ec74
I20250629 01:52:02.821635 2637 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a0b76160e22c46929936cb9682322fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b76160e22c46929936cb9682322fae" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34747 } } peers { permanent_uuid: "0f4258c004e04bcf8216bb7c8fd56c54" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 35873 } } peers { permanent_uuid: "f304077528b34c61857a659e5873631d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 32789 } } }
I20250629 01:52:02.821777 2637 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae [sys.catalog]: This master's current role is: LEADER
I20250629 01:52:02.821781 850 leader_election.cc:304] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:02.822081 850 leader_election.cc:343] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 5b8a2d1a7f7146f4968590914374ba23
I20250629 01:52:02.822155 850 leader_election.cc:304] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.822273 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0fcaee65bbdc47778737eac4a8c77fd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.822433 850 leader_election.cc:343] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: f3bdc8bda8a940be865679a05b90bd08
I20250629 01:52:02.822504 850 leader_election.cc:304] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:02.822618 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fcaee65bbdc47778737eac4a8c77fd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.822713 2831 raft_consensus.cc:2747] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.822813 2831 raft_consensus.cc:2747] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.822878 2831 raft_consensus.cc:2747] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.823535 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5d53a01d5be7418c988913852a7f91a3" } new_table_name: "default.Table_13f7043c741546969926f8ab0a5b0782" modify_external_catalogs: true
I20250629 01:52:02.823688 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f47c00ffce0403692345454e7fa9225" } modify_external_catalogs: true
I20250629 01:52:02.823793 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f47c00ffce0403692345454e7fa9225" } modify_external_catalogs: true
I20250629 01:52:02.824220 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_19a73abe17d0485088732b0719e69baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.824568 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19a73abe17d0485088732b0719e69baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.826102 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2698ae66abbc4163beb2f3939554be2a" } modify_external_catalogs: true
I20250629 01:52:02.826217 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2698ae66abbc4163beb2f3939554be2a" } modify_external_catalogs: true
I20250629 01:52:02.823535 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_99304b2c156d4784ba9cb3aed7494313"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:02.827008 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99304b2c156d4784ba9cb3aed7494313 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:02.828862 1155 tablet_bootstrap.cc:492] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.829036 1155 ts_tablet_manager.cc:1397] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.000s
I20250629 01:52:02.829838 1155 raft_consensus.cc:357] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.830063 1155 raft_consensus.cc:383] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.830157 1155 raft_consensus.cc:738] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.830399 1155 consensus_queue.cc:260] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.838568 2840 raft_consensus.cc:491] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.838706 2840 raft_consensus.cc:513] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.839161 2840 leader_election.cc:290] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:02.839679 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 81653us
W20250629 01:52:02.839943 712 consensus_peers.cc:489] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.840543 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a8a9bc691e4b158276f34a51c0dc61" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:02.840948 710 leader_election.cc:343] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:52:02.841116 2649 raft_consensus.cc:491] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.841238 2649 raft_consensus.cc:513] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.841807 2649 leader_election.cc:290] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.842268 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e2c70221e94923b6a324a03f7133c6" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.842411 1051 raft_consensus.cc:2466] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
W20250629 01:52:02.842438 712 consensus_peers.cc:489] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.842749 710 leader_election.cc:304] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.843243 2649 raft_consensus.cc:2802] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.843323 2649 raft_consensus.cc:491] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.843389 2649 raft_consensus.cc:3058] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.844717 1154 tablet_bootstrap.cc:492] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.844834 1154 ts_tablet_manager.cc:1397] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.003s
I20250629 01:52:02.845369 1154 raft_consensus.cc:357] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.845510 1154 raft_consensus.cc:383] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.845573 1154 raft_consensus.cc:738] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.845727 1154 consensus_queue.cc:260] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.846276 2649 raft_consensus.cc:513] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.846812 1154 ts_tablet_manager.cc:1428] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:02.847054 1154 tablet_bootstrap.cc:492] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.847110 2649 leader_election.cc:290] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.847049 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e2c70221e94923b6a324a03f7133c6" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:02.847721 1051 raft_consensus.cc:3058] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:02.847995 710 consensus_peers.cc:489] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ba5887dffc1460a89208447ecc6a198. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.848227 1155 ts_tablet_manager.cc:1428] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.019s user 0.001s sys 0.002s
I20250629 01:52:02.848460 1155 tablet_bootstrap.cc:492] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.849565 1154 tablet_bootstrap.cc:654] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.849684 1155 tablet_bootstrap.cc:654] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.850160 1051 raft_consensus.cc:2466] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:02.850483 710 leader_election.cc:304] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.851166 2847 raft_consensus.cc:2802] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:02.851315 2847 raft_consensus.cc:695] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:02.851537 2847 consensus_queue.cc:237] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.855031 712 consensus_peers.cc:489] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:02.855333 712 leader_election.cc:343] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.855456 712 leader_election.cc:304] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.855883 2649 raft_consensus.cc:2747] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:02.858642 712 consensus_peers.cc:489] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.859066 1154 tablet_bootstrap.cc:492] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:02.859171 1154 ts_tablet_manager.cc:1397] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
W20250629 01:52:02.859687 712 leader_election.cc:343] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.859884 1154 raft_consensus.cc:357] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.860296 1022 tablet_service.cc:1468] Processing CreateTablet for tablet f6d9b52cf8eb4a9c8840705ff1d313c8 (DEFAULT_TABLE table=default.Table_05a1132bda094159a72bb34e6b003609 [id=a765620d216b4c1788af5f10c5317424]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.860611 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d9b52cf8eb4a9c8840705ff1d313c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.861088 753 tablet_service.cc:1468] Processing CreateTablet for tablet 0c8e391ad2a347e19d5bbe60613de449 (DEFAULT_TABLE table=default.Table_70c13f7893b2471387ac7400f4f69c0f [id=733c4c07a4fd4a94961c9c3e6073fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.861466 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8e391ad2a347e19d5bbe60613de449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.863358 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0c8e391ad2a347e19d5bbe60613de449 (DEFAULT_TABLE table=default.Table_70c13f7893b2471387ac7400f4f69c0f [id=733c4c07a4fd4a94961c9c3e6073fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.863356 740 tablet_service.cc:1468] Processing CreateTablet for tablet ac0164ca224e40629bfe5ec6006fba1b (DEFAULT_TABLE table=default.Table_3f88c1cdd331456683f3fe4d95aa18af [id=7d61b2b2d3ca4bfaacacdd8ed807c12c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.863662 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0164ca224e40629bfe5ec6006fba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.863660 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8e391ad2a347e19d5bbe60613de449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.867174 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c8ad1108c6f94621a65cd5d9e3c512fc (DEFAULT_TABLE table=default.Table_6f957d68e3dd4ba9a5bf56cc925385bc [id=62b4d4b802994dcdb00c3d25f3eb8da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.867434 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ad1108c6f94621a65cd5d9e3c512fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:02.868046 712 leader_election.cc:343] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.868340 744 tablet_service.cc:1468] Processing CreateTablet for tablet c8ad1108c6f94621a65cd5d9e3c512fc (DEFAULT_TABLE table=default.Table_6f957d68e3dd4ba9a5bf56cc925385bc [id=62b4d4b802994dcdb00c3d25f3eb8da9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:02.868396 712 consensus_peers.cc:489] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47cbc958eca14ed69fb45c3c2984aeef. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.868669 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ad1108c6f94621a65cd5d9e3c512fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:02.868980 710 consensus_peers.cc:489] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6fa908ded8240bb962674e1efb29c74. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:02.860497 1156 ts_tablet_manager.cc:1918] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.869608 1156 log.cc:1199] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/8295e8178eff4d5eb95ac885092731e0
I20250629 01:52:02.869782 1155 tablet_bootstrap.cc:492] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.869867 1156 ts_tablet_manager.cc:1939] T 8295e8178eff4d5eb95ac885092731e0 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:02.869961 745 tablet_service.cc:1468] Processing CreateTablet for tablet f6d9b52cf8eb4a9c8840705ff1d313c8 (DEFAULT_TABLE table=default.Table_05a1132bda094159a72bb34e6b003609 [id=a765620d216b4c1788af5f10c5317424]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.870254 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d9b52cf8eb4a9c8840705ff1d313c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.871492 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b407e66d26684fada61d72b8e41f6800 (DEFAULT_TABLE table=default.Table_6beeaafab09745f49b66734e77f01269 [id=25ec7692197044a79de5f4b0c93a6008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.871747 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b407e66d26684fada61d72b8e41f6800. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.871821 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1374) took 1692 ms. Trace:
I20250629 01:52:02.871907 1156 rpcz_store.cc:276] 0629 01:52:01.179740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.179840 (+ 100us) service_pool.cc:225] Handling call
0629 01:52:02.805756 (+1625916us) ts_tablet_manager.cc:1145] Deleting tablet 8295e8178eff4d5eb95ac885092731e0
0629 01:52:02.805764 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:02.825691 (+ 19927us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.860372 (+ 34681us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.871801 (+ 11429us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24,"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":1625822}
I20250629 01:52:02.872939 1159 tablet_bootstrap.cc:492] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.873046 1159 ts_tablet_manager.cc:1397] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.180s user 0.008s sys 0.018s
I20250629 01:52:02.869875 1155 ts_tablet_manager.cc:1397] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s
I20250629 01:52:02.873660 1159 raft_consensus.cc:357] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:02.873713 1155 raft_consensus.cc:357] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.873792 1159 raft_consensus.cc:383] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.873824 1155 raft_consensus.cc:383] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.873829 1159 raft_consensus.cc:738] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.873867 1155 raft_consensus.cc:738] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.874003 1159 consensus_queue.cc:260] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:02.875034 2651 raft_consensus.cc:491] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.875128 2651 raft_consensus.cc:513] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.875535 2651 leader_election.cc:290] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.875784 2794 tablet.cc:1722] T 3b55c66136ba43d7b535750e04b72308 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:02.876295 751 tablet_service.cc:1468] Processing CreateTablet for tablet b407e66d26684fada61d72b8e41f6800 (DEFAULT_TABLE table=default.Table_6beeaafab09745f49b66734e77f01269 [id=25ec7692197044a79de5f4b0c93a6008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.876602 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b407e66d26684fada61d72b8e41f6800. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.878213 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a8a9bc691e4b158276f34a51c0dc61" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.878760 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "898b49812ece4b57a1a1015d9f26be2f" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.879669 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 44791a8df82546f3bde5a81d329847ca (DEFAULT_TABLE table=default.Table_70cddd5986ff4223b20b1e8d07c63456 [id=da0a3f5b89ff4506a3bb8656049dd44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.879984 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44791a8df82546f3bde5a81d329847ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.880672 1159 ts_tablet_manager.cc:1428] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250629 01:52:02.881100 1156 tablet_replica.cc:331] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:02.881309 1156 raft_consensus.cc:2241] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:02.881444 1156 raft_consensus.cc:2270] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:02.881738 1159 tablet_bootstrap.cc:492] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:02.881959 1156 ts_tablet_manager.cc:1905] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.883095 1159 tablet_bootstrap.cc:654] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.883741 757 tablet_service.cc:1468] Processing CreateTablet for tablet 44791a8df82546f3bde5a81d329847ca (DEFAULT_TABLE table=default.Table_70cddd5986ff4223b20b1e8d07c63456 [id=da0a3f5b89ff4506a3bb8656049dd44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.884004 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44791a8df82546f3bde5a81d329847ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.884542 2834 raft_consensus.cc:491] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.884635 2834 raft_consensus.cc:513] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.885090 2834 leader_election.cc:290] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.885707 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 80a9aa22f03042c7b0a2d9a486cf21ea (DEFAULT_TABLE table=default.Table_13e1b0a3c2bc48b8ae300865cb64d7b8 [id=d2e6e230d0d94e9db6063f24565cc87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.886029 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a9aa22f03042c7b0a2d9a486cf21ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.886323 740 tablet_service.cc:1468] Processing CreateTablet for tablet c051e307f30b41dbb826a003c909325e (DEFAULT_TABLE table=default.Table_9683e9406d2840bfbec404229f77ac50 [id=221821cfd9e044a29e64e02d8909ec77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.886607 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c051e307f30b41dbb826a003c909325e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.887683 2670 raft_consensus.cc:491] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.887789 2670 raft_consensus.cc:513] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.888334 1021 tablet_service.cc:1468] Processing CreateTablet for tablet ea0c5b50fc0c4446873a771d8a8e7603 (DEFAULT_TABLE table=default.Table_2ebb3eb6a01f4d388ddaa95cf3cfad0f [id=406a2593c3274d6bac8918071c3790f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.888573 2670 leader_election.cc:290] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:02.888654 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0c5b50fc0c4446873a771d8a8e7603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.892323 750 tablet_service.cc:1468] Processing CreateTablet for tablet ea0c5b50fc0c4446873a771d8a8e7603 (DEFAULT_TABLE table=default.Table_2ebb3eb6a01f4d388ddaa95cf3cfad0f [id=406a2593c3274d6bac8918071c3790f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.879282 1155 consensus_queue.cc:260] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.892565 2837 raft_consensus.cc:491] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.892652 2837 raft_consensus.cc:513] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.892764 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ac0164ca224e40629bfe5ec6006fba1b (DEFAULT_TABLE table=default.Table_3f88c1cdd331456683f3fe4d95aa18af [id=7d61b2b2d3ca4bfaacacdd8ed807c12c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.893023 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0164ca224e40629bfe5ec6006fba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.893388 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8635903caac9480184c32e6689870485" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:02.893420 2837 leader_election.cc:290] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.893522 1048 raft_consensus.cc:2466] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:02.893728 742 tablet_service.cc:1468] Processing CreateTablet for tablet 80a9aa22f03042c7b0a2d9a486cf21ea (DEFAULT_TABLE table=default.Table_13e1b0a3c2bc48b8ae300865cb64d7b8 [id=d2e6e230d0d94e9db6063f24565cc87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.893790 710 leader_election.cc:304] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:02.893967 2837 raft_consensus.cc:2802] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.894016 2837 raft_consensus.cc:491] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:02.894078 2837 raft_consensus.cc:3058] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.896853 2837 raft_consensus.cc:513] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.898955 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8635903caac9480184c32e6689870485" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:02.899039 710 leader_election.cc:343] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 898b49812ece4b57a1a1015d9f26be2f
I20250629 01:52:02.899107 774 raft_consensus.cc:2391] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:52:02.899541 916 raft_consensus.cc:3058] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.900316 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e2c70221e94923b6a324a03f7133c6" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.901343 753 tablet_service.cc:1468] Processing CreateTablet for tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec (DEFAULT_TABLE table=default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.902009 756 tablet_service.cc:1468] Processing CreateTablet for tablet d60a756dd2ed4a2e96aaf3f13e3a90de (DEFAULT_TABLE table=default.Table_13401ad7d0e2429a8e31aff8e7a61a3e [id=3abcef6a2cb246c881ce292d1e05b5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.902355 1155 ts_tablet_manager.cc:1428] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.029s user 0.001s sys 0.000s
I20250629 01:52:02.901175 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e2c70221e94923b6a324a03f7133c6" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:02.892573 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0c5b50fc0c4446873a771d8a8e7603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.902868 1155 tablet_bootstrap.cc:492] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:02.903028 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.903084 743 tablet_service.cc:1468] Processing CreateTablet for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5 (DEFAULT_TABLE table=default.Table_67461ab444b64f80854e45ddd5e4bcb8 [id=4b5873cf4af74847a7940a36f395927c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.903352 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.905544 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60a756dd2ed4a2e96aaf3f13e3a90de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.908514 2837 leader_election.cc:290] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:02.909271 712 leader_election.cc:343] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.909348 712 leader_election.cc:304] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.909523 2651 raft_consensus.cc:2747] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:02.910131 712 leader_election.cc:343] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.910202 712 leader_election.cc:304] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:02.910351 2651 raft_consensus.cc:2747] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:02.910912 712 consensus_peers.cc:489] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6fa908ded8240bb962674e1efb29c74. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:02.911448 712 leader_election.cc:343] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.914144 1155 tablet_bootstrap.cc:654] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:02.914556 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8635903caac9480184c32e6689870485" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.915151 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8635903caac9480184c32e6689870485" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:02.916455 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a9aa22f03042c7b0a2d9a486cf21ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.926204 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 8295e8178eff4d5eb95ac885092731e0 (table default.Table_0d8a47c941e64eb1bd5332c0dbee8504 [id=c17c3600bb2447d69097c8223ce973f3]) successfully deleted
I20250629 01:52:02.935187 1156 ts_tablet_manager.cc:1918] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:52:02.937053 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 114611us
I20250629 01:52:02.938158 2855 tablet.cc:1722] T 3b55c66136ba43d7b535750e04b72308 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:02.939311 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e2c70221e94923b6a324a03f7133c6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.939860 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1ebbcb2c26b4667b2405154a9d8fccb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.941038 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941334 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9370ec79715e4e5fa4766f202765f04b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941366 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ac9d77381f7e4241aa546eaea3077168 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941478 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 72b2caf969db48458ea007ae03a0640f with delete_type TABLET_DATA_DELETED (replaced by tablet 28bc247520074b70973312e183ed6796) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941509 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 412a3d30c85442cd943a163cf5ea61b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 08dd5ec4c05f4408a3b832161199140b) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941625 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 819ed44d98f14075b9a67b88dc44e86e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941535 883 tablet_service.cc:1515] Processing DeleteTablet for tablet daf312e60c014fe698d3ec9f1a1b769c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941883 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 3a81e6f9711c4d398ae09f33a75566ae with delete_type TABLET_DATA_DELETED (replaced by tablet 60732c68906b4805bfa63cc4dafb4ceb) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.941955 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d9ef77181ba48f3a6de16b945c8df1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.942054 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 06e31520d3ad4c3395a59049755a4725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:53 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:02.942303 895 tablet_service.cc:1468] Processing CreateTablet for tablet 0c8e391ad2a347e19d5bbe60613de449 (DEFAULT_TABLE table=default.Table_70c13f7893b2471387ac7400f4f69c0f [id=733c4c07a4fd4a94961c9c3e6073fe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942423 886 tablet_service.cc:1468] Processing CreateTablet for tablet b407e66d26684fada61d72b8e41f6800 (DEFAULT_TABLE table=default.Table_6beeaafab09745f49b66734e77f01269 [id=25ec7692197044a79de5f4b0c93a6008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942454 878 tablet_service.cc:1468] Processing CreateTablet for tablet c8ad1108c6f94621a65cd5d9e3c512fc (DEFAULT_TABLE table=default.Table_6f957d68e3dd4ba9a5bf56cc925385bc [id=62b4d4b802994dcdb00c3d25f3eb8da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942605 877 tablet_service.cc:1468] Processing CreateTablet for tablet 44791a8df82546f3bde5a81d329847ca (DEFAULT_TABLE table=default.Table_70cddd5986ff4223b20b1e8d07c63456 [id=da0a3f5b89ff4506a3bb8656049dd44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942698 881 tablet_service.cc:1468] Processing CreateTablet for tablet ac0164ca224e40629bfe5ec6006fba1b (DEFAULT_TABLE table=default.Table_3f88c1cdd331456683f3fe4d95aa18af [id=7d61b2b2d3ca4bfaacacdd8ed807c12c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942713 888 tablet_service.cc:1468] Processing CreateTablet for tablet 80a9aa22f03042c7b0a2d9a486cf21ea (DEFAULT_TABLE table=default.Table_13e1b0a3c2bc48b8ae300865cb64d7b8 [id=d2e6e230d0d94e9db6063f24565cc87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942878 892 tablet_service.cc:1468] Processing CreateTablet for tablet c051e307f30b41dbb826a003c909325e (DEFAULT_TABLE table=default.Table_9683e9406d2840bfbec404229f77ac50 [id=221821cfd9e044a29e64e02d8909ec77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942904 880 tablet_service.cc:1468] Processing CreateTablet for tablet d60a756dd2ed4a2e96aaf3f13e3a90de (DEFAULT_TABLE table=default.Table_13401ad7d0e2429a8e31aff8e7a61a3e [id=3abcef6a2cb246c881ce292d1e05b5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.942898 885 tablet_service.cc:1468] Processing CreateTablet for tablet ea0c5b50fc0c4446873a771d8a8e7603 (DEFAULT_TABLE table=default.Table_2ebb3eb6a01f4d388ddaa95cf3cfad0f [id=406a2593c3274d6bac8918071c3790f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.943065 891 tablet_service.cc:1468] Processing CreateTablet for tablet 52ab2b9f2d34438e86fec9a100f6671d (DEFAULT_TABLE table=default.Table_096d8cb0cba44f5db475e2badfbea1b0 [id=0e02c8247ac049cf849374e01ac570fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.943346 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec (DEFAULT_TABLE table=default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:02.943361 850 leader_election.cc:343] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 4d9ef77181ba48f3a6de16b945c8df1a
I20250629 01:52:02.943444 879 tablet_service.cc:1468] Processing CreateTablet for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5 (DEFAULT_TABLE table=default.Table_67461ab444b64f80854e45ddd5e4bcb8 [id=4b5873cf4af74847a7940a36f395927c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.943552 850 leader_election.cc:304] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:02.942303 883 tablet_service.cc:1468] Processing CreateTablet for tablet f6d9b52cf8eb4a9c8840705ff1d313c8 (DEFAULT_TABLE table=default.Table_05a1132bda094159a72bb34e6b003609 [id=a765620d216b4c1788af5f10c5317424]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.944067 2845 raft_consensus.cc:2747] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:02.944615 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67e90a48af8f48bba3b3fd4f4110d367" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:02.944787 915 raft_consensus.cc:3058] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.945832 1159 tablet_bootstrap.cc:492] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:02.946230 1159 ts_tablet_manager.cc:1397] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.002s
I20250629 01:52:02.946465 913 raft_consensus.cc:3058] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.947991 1159 raft_consensus.cc:357] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.948346 1159 raft_consensus.cc:383] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.948565 1159 raft_consensus.cc:738] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.948359 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b407e66d26684fada61d72b8e41f6800. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.949012 1159 consensus_queue.cc:260] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.950430 2670 raft_consensus.cc:491] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:02.950539 2670 raft_consensus.cc:513] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:02.950977 2670 leader_election.cc:290] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:02.952200 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.956182 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ab2b9f2d34438e86fec9a100f6671d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.860023 1154 raft_consensus.cc:383] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:02.959359 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44791a8df82546f3bde5a81d329847ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.965606 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6d9b52cf8eb4a9c8840705ff1d313c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.901175 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 52ab2b9f2d34438e86fec9a100f6671d (DEFAULT_TABLE table=default.Table_096d8cb0cba44f5db475e2badfbea1b0 [id=0e02c8247ac049cf849374e01ac570fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.967159 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ab2b9f2d34438e86fec9a100f6671d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.968305 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c8e391ad2a347e19d5bbe60613de449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.901582 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec (DEFAULT_TABLE table=default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.969779 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2eeb87e0e90a4dd4a7bfbb1f60e266ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.972256 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8ad1108c6f94621a65cd5d9e3c512fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.901834 1013 tablet_service.cc:1468] Processing CreateTablet for tablet d60a756dd2ed4a2e96aaf3f13e3a90de (DEFAULT_TABLE table=default.Table_13401ad7d0e2429a8e31aff8e7a61a3e [id=3abcef6a2cb246c881ce292d1e05b5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.972576 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60a756dd2ed4a2e96aaf3f13e3a90de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.976262 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0164ca224e40629bfe5ec6006fba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.980275 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a9aa22f03042c7b0a2d9a486cf21ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.980638 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d60a756dd2ed4a2e96aaf3f13e3a90de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.980954 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0c5b50fc0c4446873a771d8a8e7603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.981644 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c051e307f30b41dbb826a003c909325e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.982070 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.902711 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5 (DEFAULT_TABLE table=default.Table_67461ab444b64f80854e45ddd5e4bcb8 [id=4b5873cf4af74847a7940a36f395927c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.988168 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.900305 1158 ts_tablet_manager.cc:1918] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:02.888335 1018 tablet_service.cc:1468] Processing CreateTablet for tablet c051e307f30b41dbb826a003c909325e (DEFAULT_TABLE table=default.Table_9683e9406d2840bfbec404229f77ac50 [id=221821cfd9e044a29e64e02d8909ec77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:02.992419 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c051e307f30b41dbb826a003c909325e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:02.992893 1158 log.cc:1199] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/d1a8a9bc691e4b158276f34a51c0dc61
W20250629 01:52:02.996176 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 999dae3757ad4c57bf4e2a92ee5203ca consensus state has no leader
I20250629 01:52:02.911285 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8635903caac9480184c32e6689870485" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:02.996276 1052 raft_consensus.cc:3058] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:02.996508 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5cd8e243b623461eabaa41b2b6d9988a" } new_table_name: "default.Table_40cce4204c6c494496b9b935256d8aae" modify_external_catalogs: true
I20250629 01:52:02.998530 1052 raft_consensus.cc:2466] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:02.943990 988 leader_election.cc:343] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:02.905586 2852 tablet.cc:1722] T 3b55c66136ba43d7b535750e04b72308 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:52:02.998816 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 54835.2us
I20250629 01:52:02.959239 1154 raft_consensus.cc:738] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:02.999034 1154 consensus_queue.cc:260] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:02.902011 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 121009us
I20250629 01:52:02.999363 986 leader_election.cc:304] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:02.999512 1154 ts_tablet_manager.cc:1428] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.140s user 0.001s sys 0.001s
I20250629 01:52:02.999778 1154 tablet_bootstrap.cc:492] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:02.999893 2764 raft_consensus.cc:2802] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:02.999953 2764 raft_consensus.cc:491] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.000003 2764 raft_consensus.cc:3058] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.001253 1154 tablet_bootstrap.cc:654] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.002262 2764 raft_consensus.cc:513] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:02.999765 986 leader_election.cc:304] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.002380 449 catalog_manager.cc:5582] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.002674 2758 raft_consensus.cc:2802] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.002737 2764 leader_election.cc:290] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.002780 2758 raft_consensus.cc:695] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:03.002976 2758 consensus_queue.cc:237] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.003008 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c07a1580c734aeba695151d23c7e020 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:03.006023 1154 tablet_bootstrap.cc:492] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.006130 1154 ts_tablet_manager.cc:1397] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250629 01:52:03.006686 1154 raft_consensus.cc:357] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.006825 1154 raft_consensus.cc:383] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.006879 1154 raft_consensus.cc:738] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.007030 1154 consensus_queue.cc:260] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.003647 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b4fbc288180d41e2be6faef2985f4b0e" } new_table_name: "default.Table_afc4ba92b5ec4436bb59d740a0039468" modify_external_catalogs: true
I20250629 01:52:03.007822 986 leader_election.cc:304] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.008167 1154 ts_tablet_manager.cc:1428] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:03.004355 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3a7fe7d34c6e4d919869404da9508244" } new_table_name: "default.Table_de381987e97441598327a358420b84a9" modify_external_catalogs: true
I20250629 01:52:03.009024 1154 tablet_bootstrap.cc:492] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.005035 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0198d8f0bc1443ac86dbfa3f74614a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.010630 1154 tablet_bootstrap.cc:654] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:03.010833 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0198d8f0bc1443ac86dbfa3f74614a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.012517 449 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:03.015368 2760 raft_consensus.cc:2747] T d062fb830020442faee626be006926c1 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.015751 986 leader_election.cc:304] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.015986 2760 raft_consensus.cc:2802] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.016042 2760 raft_consensus.cc:491] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.016112 2760 raft_consensus.cc:3058] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.018265 2760 raft_consensus.cc:513] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.018704 2760 leader_election.cc:290] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.023533 1158 ts_tablet_manager.cc:1939] T d1a8a9bc691e4b158276f34a51c0dc61 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.024276 1158 tablet_replica.cc:331] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.024451 1158 raft_consensus.cc:2241] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.027522 1159 ts_tablet_manager.cc:1428] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.081s user 0.000s sys 0.002s
I20250629 01:52:03.027757 1159 tablet_bootstrap.cc:492] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.028972 1159 tablet_bootstrap.cc:654] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.032277 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet d1a8a9bc691e4b158276f34a51c0dc61 (table default.Table_a968452543bd46baa06280c24819ada1 [id=a9ff1c784a004f5cad1cf1a1c8066d8d]) successfully deleted
I20250629 01:52:03.024571 1158 raft_consensus.cc:2270] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.033591 1158 ts_tablet_manager.cc:1905] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.037524 1158 ts_tablet_manager.cc:1918] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.037606 1158 log.cc:1199] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/7cc18d1154ef4d05b3b82180a9c1ca8b
I20250629 01:52:03.037847 1158 ts_tablet_manager.cc:1939] T 7cc18d1154ef4d05b3b82180a9c1ca8b P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.038887 1158 tablet_replica.cc:331] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.039021 1158 raft_consensus.cc:2241] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.040628 1158 raft_consensus.cc:2270] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.040905 1159 tablet_bootstrap.cc:492] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.041010 1159 ts_tablet_manager.cc:1397] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250629 01:52:03.041038 1158 ts_tablet_manager.cc:1905] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.040629 986 leader_election.cc:304] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.041642 1159 raft_consensus.cc:357] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.041774 1159 raft_consensus.cc:383] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.041821 1159 raft_consensus.cc:738] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.041963 1159 consensus_queue.cc:260] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.042309 2760 raft_consensus.cc:2747] T 8635903caac9480184c32e6689870485 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.044503 2760 raft_consensus.cc:491] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.044607 2760 raft_consensus.cc:513] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.044677 2859 tablet_replica.cc:331] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:03.044836 2859 raft_consensus.cc:2241] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.044934 2859 raft_consensus.cc:2270] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.045045 2760 leader_election.cc:290] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.045117 1159 ts_tablet_manager.cc:1428] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250629 01:52:03.045367 2764 raft_consensus.cc:491] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.045482 2764 raft_consensus.cc:513] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.045691 2859 ts_tablet_manager.cc:1905] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.045933 2764 leader_election.cc:290] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.046307 2764 raft_consensus.cc:491] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.046456 1159 tablet_bootstrap.cc:492] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.046399 2764 raft_consensus.cc:513] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.046789 2764 leader_election.cc:290] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.046877 2764 raft_consensus.cc:491] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.046948 2764 raft_consensus.cc:513] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.047363 2764 leader_election.cc:290] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.049571 1159 tablet_bootstrap.cc:654] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.054790 1158 ts_tablet_manager.cc:1918] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.055516 2670 raft_consensus.cc:491] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.055619 2670 raft_consensus.cc:513] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.055727 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 7cc18d1154ef4d05b3b82180a9c1ca8b (table default.Table_9a4eca53d6d5404fa56df4720474d29c [id=e61ee06c5a0343ecb060fea35f151e6c]) successfully deleted
W20250629 01:52:03.055899 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 412a3d30c85442cd943a163cf5ea61b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 412a3d30c85442cd943a163cf5ea61b5 already in progress: creating tablet
I20250629 01:52:03.056038 2670 leader_election.cc:290] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.056124 1158 log.cc:1199] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/97d0fbb691514d51a1ca71f32a666c2d
I20250629 01:52:03.058128 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1ac60a024a074953b513ae4f60840d25
I20250629 01:52:03.058552 455 catalog_manager.cc:6652] ReplaceTablet: tablet 1ac60a024a074953b513ae4f60840d25 deleted and replaced by tablet 2bd218fc4ab94adb8a1680e84bb88c00
I20250629 01:52:03.059073 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 12873ec9a5574e789f9d669ddda85704
I20250629 01:52:03.059790 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4416bbec7d1c40df9a835b503d368d4c
I20250629 01:52:03.061645 2834 raft_consensus.cc:491] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.061756 2834 raft_consensus.cc:513] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.062340 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ac60a024a074953b513ae4f60840d25 with delete_type TABLET_DATA_DELETED (replaced by tablet 2bd218fc4ab94adb8a1680e84bb88c00) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.062713 2834 leader_election.cc:290] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.062923 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 12873ec9a5574e789f9d669ddda85704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:03.063663 988 leader_election.cc:343] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.064128 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4416bbec7d1c40df9a835b503d368d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:03.064369 988 leader_election.cc:343] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.067303 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 already in progress: creating tablet
W20250629 01:52:03.067564 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 9370ec79715e4e5fa4766f202765f04b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9370ec79715e4e5fa4766f202765f04b already in progress: creating tablet
I20250629 01:52:03.068078 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d0fbb691514d51a1ca71f32a666c2d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.068300 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12b85f220f3947ca8110f2db937e7b68" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.068404 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43b845c3e7174618bd5c394d7206a634" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.068665 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ec92eddfabf40918a6cabaff5fe113a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.068697 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a75d7986152f427d88ed34a443128bc6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.068781 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 12873ec9a5574e789f9d669ddda85704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.068404 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b6d5ff92c343cb8f1b3039b84390af" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.069054 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 4416bbec7d1c40df9a835b503d368d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.069115 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.069188 850 leader_election.cc:343] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 819ed44d98f14075b9a67b88dc44e86e
W20250629 01:52:03.069290 850 leader_election.cc:343] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 3a81e6f9711c4d398ae09f33a75566ae
W20250629 01:52:03.069370 850 leader_election.cc:343] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 9fe9674ffeb44ba6ab60de59d4c08670
W20250629 01:52:03.069545 988 leader_election.cc:343] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.069813 988 leader_election.cc:343] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.070046 988 leader_election.cc:343] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.070271 988 leader_election.cc:343] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.070500 988 leader_election.cc:343] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.070722 988 leader_election.cc:343] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.070945 988 leader_election.cc:343] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.071871 1158 ts_tablet_manager.cc:1939] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.072170 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ac60a024a074953b513ae4f60840d25 with delete_type TABLET_DATA_DELETED (replaced by tablet 2bd218fc4ab94adb8a1680e84bb88c00) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.072538 1158 tablet_replica.cc:331] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.072575 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 97d0fbb691514d51a1ca71f32a666c2d (table default.Table_30b8e1c0bc8241a7ba265c3c4f86a095 [id=5ad43132e07d4ddeb9d1716e8e314256]) successfully deleted
I20250629 01:52:03.072710 1158 raft_consensus.cc:2241] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250629 01:52:03.072752 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2692a0c5508d4f12b3f8f784f1f1bbd4 already in progress: creating tablet
I20250629 01:52:03.072834 1158 raft_consensus.cc:2270] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.075070 1158 ts_tablet_manager.cc:1905] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:02.991029 740 tablet_service.cc:1468] Processing CreateTablet for tablet 52ab2b9f2d34438e86fec9a100f6671d (DEFAULT_TABLE table=default.Table_096d8cb0cba44f5db475e2badfbea1b0 [id=0e02c8247ac049cf849374e01ac570fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.081465 1156 log.cc:1199] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:52:03.081777 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ab2b9f2d34438e86fec9a100f6671d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.081810 1156 ts_tablet_manager.cc:1939] T 327c303b6b9347f5b4cb78f74d41a44a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.082463 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1370) took 1903 ms. Trace:
I20250629 01:52:03.082535 1156 rpcz_store.cc:276] 0629 01:52:01.178726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.179986 (+ 1260us) service_pool.cc:225] Handling call
0629 01:52:02.881050 (+1701064us) ts_tablet_manager.cc:1145] Deleting tablet 327c303b6b9347f5b4cb78f74d41a44a
0629 01:52:02.881061 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:02.932479 (+ 51418us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:02.935128 (+ 2649us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.082449 (+147321us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1700910}
I20250629 01:52:03.082840 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1366) took 1913 ms. Trace:
I20250629 01:52:03.082887 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 327c303b6b9347f5b4cb78f74d41a44a (table default.Table_209b7bc3f1de4919a043f2ab3c1c5d6a [id=d62355a430724b96abb294c451c9bd58]) successfully deleted
I20250629 01:52:03.082894 1156 rpcz_store.cc:276] 0629 01:52:01.169683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.187046 (+ 17363us) service_pool.cc:225] Handling call
0629 01:52:03.082753 (+1895707us) ts_tablet_manager.cc:1145] Deleting tablet 2045012c80d049eaabf7dec1df28b087
0629 01:52:03.082760 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.082829 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1895540}
I20250629 01:52:03.083101 1156 tablet_replica.cc:331] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:52:03.083205 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 2045012c80d049eaabf7dec1df28b087 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2045012c80d049eaabf7dec1df28b087 already in progress: creating tablet
I20250629 01:52:03.083230 1156 raft_consensus.cc:2241] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:03.083432 1156 raft_consensus.cc:2270] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.083815 2834 raft_consensus.cc:491] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.083853 1156 ts_tablet_manager.cc:1905] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.083907 2834 raft_consensus.cc:513] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.084455 2834 leader_election.cc:290] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:02.998003 1155 tablet_bootstrap.cc:492] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:02.999248 710 leader_election.cc:304] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:03.090536 2855 tablet.cc:1722] T cd9a41fae477470281dfba4d4e552b99 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:03.092566 2864 mvcc.cc:204] Tried to move back new op lower bound from 7172759227249786880 to 7172759224041758720. Current Snapshot: MvccSnapshot[applied={T|T < 7172759227249786880}]
I20250629 01:52:03.093381 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01d95356c7a7419bb56ddf67d451eadc" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.093681 988 leader_election.cc:343] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.094278 2845 raft_consensus.cc:491] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.094377 2845 raft_consensus.cc:513] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.094897 1154 tablet_bootstrap.cc:492] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.094914 2845 leader_election.cc:290] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.095002 1154 ts_tablet_manager.cc:1397] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.003s
I20250629 01:52:03.095546 1154 raft_consensus.cc:357] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.095661 1154 raft_consensus.cc:383] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.095697 1154 raft_consensus.cc:738] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.095853 1154 consensus_queue.cc:260] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.098593 1154 ts_tablet_manager.cc:1428] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:52:03.099191 1154 tablet_bootstrap.cc:492] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:52:03.100029 850 leader_election.cc:343] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: f94d8fc1b6254ba69acadcaff3613d97
I20250629 01:52:03.100841 1154 tablet_bootstrap.cc:654] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.103694 1159 tablet_bootstrap.cc:492] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.103785 1159 ts_tablet_manager.cc:1397] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.016s
I20250629 01:52:03.104327 1159 raft_consensus.cc:357] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.104460 1159 raft_consensus.cc:383] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.104502 1159 raft_consensus.cc:738] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.104650 1159 consensus_queue.cc:260] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.105015 1159 ts_tablet_manager.cc:1428] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:03.105208 1159 tablet_bootstrap.cc:492] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.106835 1159 tablet_bootstrap.cc:654] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.110994 2859 ts_tablet_manager.cc:1918] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.111095 2859 log.cc:1199] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/ac9d77381f7e4241aa546eaea3077168
I20250629 01:52:03.111330 2859 ts_tablet_manager.cc:1939] T ac9d77381f7e4241aa546eaea3077168 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:03.111373 1156 ts_tablet_manager.cc:1918] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:03.111450 1156 log.cc:1199] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/a792a26725954adea15ad57e31bfb224
I20250629 01:52:03.111757 1156 ts_tablet_manager.cc:1939] T a792a26725954adea15ad57e31bfb224 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.111754 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet ac9d77381f7e4241aa546eaea3077168 (table default.Table_92526d1aecd94064817e09aca50dd8d0 [id=ceaecddcc74f48b4ad47d03ea91dd9d6]) successfully deleted
W20250629 01:52:03.112217 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 412a3d30c85442cd943a163cf5ea61b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 412a3d30c85442cd943a163cf5ea61b5 already in progress: creating tablet
I20250629 01:52:03.112365 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1386) took 1809 ms. Trace:
I20250629 01:52:03.112426 1156 rpcz_store.cc:276] 0629 01:52:01.303313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.305744 (+ 2431us) service_pool.cc:225] Handling call
0629 01:52:03.083072 (+1777328us) ts_tablet_manager.cc:1145] Deleting tablet a792a26725954adea15ad57e31bfb224
0629 01:52:03.083078 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.087058 (+ 3980us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.111298 (+ 24240us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.112351 (+ 1053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615377}
I20250629 01:52:03.112798 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1387) took 1809 ms. Trace:
I20250629 01:52:03.112862 1156 rpcz_store.cc:276] 0629 01:52:01.303409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.305864 (+ 2455us) service_pool.cc:225] Handling call
0629 01:52:03.112691 (+1806827us) ts_tablet_manager.cc:1145] Deleting tablet a1ddbbbe43044a70a80e7dd66011de81
0629 01:52:03.112701 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.112783 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1644829}
I20250629 01:52:03.113144 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1385) took 1809 ms. Trace:
I20250629 01:52:03.113201 1156 rpcz_store.cc:276] 0629 01:52:01.303228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.305985 (+ 2757us) service_pool.cc:225] Handling call
0629 01:52:03.113058 (+1807073us) ts_tablet_manager.cc:1145] Deleting tablet 20ca3db3bedd4ce8bd8159512a15a3e6
0629 01:52:03.113067 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.113131 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1645149}
I20250629 01:52:03.113417 1156 tablet_replica.cc:331] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:03.114825 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet a792a26725954adea15ad57e31bfb224 (table default.Table_b1e075cbb0dc4e5eb8d4de302ab08610 [id=b7fb18171e21446d99879ee1cb1cc960]) successfully deleted
W20250629 01:52:03.114984 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a1ddbbbe43044a70a80e7dd66011de81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1ddbbbe43044a70a80e7dd66011de81 already in progress: creating tablet
W20250629 01:52:03.115114 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 20ca3db3bedd4ce8bd8159512a15a3e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20ca3db3bedd4ce8bd8159512a15a3e6 already in progress: creating tablet
I20250629 01:52:03.115991 2859 tablet_replica.cc:331] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:03.116166 2859 raft_consensus.cc:2241] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.116271 2859 raft_consensus.cc:2270] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:03.116760 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 0cf193900a2842e4983a00c8a4f301a9 (table default.Table_83341ece52744f4e90689906ee72b37a [id=75722b4530384e63a406aee1edb7b237], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0cf193900a2842e4983a00c8a4f301a9 consensus state has no leader
I20250629 01:52:03.116900 2859 ts_tablet_manager.cc:1905] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.117064 1156 raft_consensus.cc:2241] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:03.117290 1156 pending_rounds.cc:70] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:03.117347 1156 pending_rounds.cc:77] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759228562518016 op_type: NO_OP noop_request { }
I20250629 01:52:03.117480 1156 raft_consensus.cc:2887] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:03.117595 1156 raft_consensus.cc:2270] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.118027 1156 ts_tablet_manager.cc:1905] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:02.935770 712 leader_election.cc:343] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.062224 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 1ac60a024a074953b513ae4f60840d25 with delete_type TABLET_DATA_DELETED (replaced by tablet 2bd218fc4ab94adb8a1680e84bb88c00) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.063264 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 12873ec9a5574e789f9d669ddda85704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.063719 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 4416bbec7d1c40df9a835b503d368d4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
W20250629 01:52:03.120419 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 184622us
I20250629 01:52:03.123172 1156 ts_tablet_manager.cc:1918] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:03.126092 1154 tablet_bootstrap.cc:492] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.126201 1154 ts_tablet_manager.cc:1397] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20250629 01:52:03.123737 1155 ts_tablet_manager.cc:1397] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.221s user 0.003s sys 0.000s
I20250629 01:52:03.126796 1154 raft_consensus.cc:357] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.126953 1154 raft_consensus.cc:383] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.127005 1154 raft_consensus.cc:738] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.126235 1156 log.cc:1199] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/50d23f589de84dd5b150a17cee00c411
I20250629 01:52:03.127192 1154 consensus_queue.cc:260] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.128302 1154 ts_tablet_manager.cc:1428] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:03.130040 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 13d779335d09442682554322b59f915a
I20250629 01:52:03.131274 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_79af1ae8dbb242a9a55e80e72378d993"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.131878 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79af1ae8dbb242a9a55e80e72378d993 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.134317 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2ed6d41c999f40d994a3609923eeffd3" } new_table_name: "default.Table_0fa540dfc0dc4e25ae2927c35c3e1efd" modify_external_catalogs: true
I20250629 01:52:03.134568 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:03.134778 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:52:03.135102 448 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:03.135354 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a90148b3c1704f9db343e6bec329546e" } modify_external_catalogs: true
I20250629 01:52:03.135474 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a90148b3c1704f9db343e6bec329546e" } modify_external_catalogs: true
I20250629 01:52:03.136389 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 13d779335d09442682554322b59f915a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.136408 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 13d779335d09442682554322b59f915a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.137965 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b8a2d1a7f7146f4968590914374ba23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:03.138478 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab8cf887e2404bd38029df6aebaa363d consensus state has no leader
I20250629 01:52:03.138525 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4c07a1580c734aeba695151d23c7e020
I20250629 01:52:03.138906 453 catalog_manager.cc:6652] ReplaceTablet: tablet 4c07a1580c734aeba695151d23c7e020 deleted and replaced by tablet 30fc706070da4952bc89b31bc5f877a1
W20250629 01:52:03.139497 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa8bc85c31eb4595905493c113fe145c consensus state has no leader
W20250629 01:52:03.139973 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 728ab44377bc44208308df54fe5366da consensus state has no leader
I20250629 01:52:03.140805 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c07a1580c734aeba695151d23c7e020 with delete_type TABLET_DATA_DELETED (replaced by tablet 30fc706070da4952bc89b31bc5f877a1) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.140973 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c07a1580c734aeba695151d23c7e020 with delete_type TABLET_DATA_DELETED (replaced by tablet 30fc706070da4952bc89b31bc5f877a1) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.142194 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 8975c1deaaf44761be791f460b53ec74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.142520 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7cf9a31f57704fea83af6665be37020e" } modify_external_catalogs: true
I20250629 01:52:03.142639 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7cf9a31f57704fea83af6665be37020e" } modify_external_catalogs: true
I20250629 01:52:03.142788 2758 raft_consensus.cc:491] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.142886 2758 raft_consensus.cc:513] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.143347 2758 leader_election.cc:290] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.144174 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bccdcd9de08648bb925db6fe56f97d20" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.145020 988 leader_election.cc:343] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.145277 1154 tablet_bootstrap.cc:492] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.145323 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 7aef2514a87c4c4aaf4d51b314630185 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:03.146836 1154 tablet_bootstrap.cc:654] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.146915 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_eb68b194087045e2886841418bd9c3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.147322 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb68b194087045e2886841418bd9c3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.147596 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ae1722f51df94fb390a076f2139a7e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.147974 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae1722f51df94fb390a076f2139a7e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.148265 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_73d48f855413440183cc015c9e93636c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.148625 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73d48f855413440183cc015c9e93636c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.150981 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.151121 880 tablet_service.cc:1515] Processing DeleteTablet for tablet f3bdc8bda8a940be865679a05b90bd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.151242 892 tablet_service.cc:1515] Processing DeleteTablet for tablet f94d8fc1b6254ba69acadcaff3613d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.151295 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b0782c7e652419096710563930f2072 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.151355 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 45ca747f03524076b4112e4db78614fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.151454 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 9fe9674ffeb44ba6ab60de59d4c08670 with delete_type TABLET_DATA_DELETED (replaced by tablet f465d0bb46674763abfc8d93f45cd13d) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.133936 757 tablet_service.cc:1515] Processing DeleteTablet for tablet 13d779335d09442682554322b59f915a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:02 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.151643 2859 ts_tablet_manager.cc:1918] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.143021 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c07a1580c734aeba695151d23c7e020 with delete_type TABLET_DATA_DELETED (replaced by tablet 30fc706070da4952bc89b31bc5f877a1) from {username='slave'} at 127.0.0.1:38530
W20250629 01:52:03.144553 712 leader_election.cc:343] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.144703 2548 raft_consensus.cc:2802] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.151873 2548 raft_consensus.cc:695] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:03.152087 2548 consensus_queue.cc:237] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.154641 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7e2c70221e94923b6a324a03f7133c6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.155499 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1ebbcb2c26b4667b2405154a9d8fccb" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:03.155908 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5712d48e7ce49eea600bbc62620c46c consensus state has no leader
I20250629 01:52:03.156232 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67e90a48af8f48bba3b3fd4f4110d367" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.156618 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d0fbb691514d51a1ca71f32a666c2d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.156733 774 raft_consensus.cc:3058] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.157006 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 790a1347e57f4cea895132564eafae9c
I20250629 01:52:03.158320 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 790a1347e57f4cea895132564eafae9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.159066 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 790a1347e57f4cea895132564eafae9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.159062 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 790a1347e57f4cea895132564eafae9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.159255 774 raft_consensus.cc:2466] T 97d0fbb691514d51a1ca71f32a666c2d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:03.152077 1156 ts_tablet_manager.cc:1939] T 50d23f589de84dd5b150a17cee00c411 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.161209 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1391) took 1623 ms. Trace:
I20250629 01:52:03.162147 1156 rpcz_store.cc:276] 0629 01:52:01.537637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.537675 (+ 38us) service_pool.cc:225] Handling call
0629 01:52:03.113382 (+1575707us) ts_tablet_manager.cc:1145] Deleting tablet 50d23f589de84dd5b150a17cee00c411
0629 01:52:03.113390 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.120699 (+ 7309us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.123109 (+ 2410us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.161190 (+ 38081us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3491,"tablet-delete.queue_time_us":1575571}
I20250629 01:52:03.162921 1156 tablet_replica.cc:331] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:03.163326 1156 raft_consensus.cc:2241] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:03.164462 2845 raft_consensus.cc:491] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.164572 2845 raft_consensus.cc:513] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.164970 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 50d23f589de84dd5b150a17cee00c411 (table default.Table_56ce186b15474e3ebf35ac8f454778bf [id=5a9ebb3e7cd54996a1d7e312bcc3d8fb]) successfully deleted
I20250629 01:52:03.165030 2845 leader_election.cc:290] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:03.165843 850 leader_election.cc:343] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:52:03.166365 1156 pending_rounds.cc:70] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:03.166643 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43b845c3e7174618bd5c394d7206a634" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.166777 774 raft_consensus.cc:3058] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.166656 1156 pending_rounds.cc:77] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759229049610240 op_type: NO_OP noop_request { }
I20250629 01:52:03.166937 1156 raft_consensus.cc:2887] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:03.167106 1156 raft_consensus.cc:2270] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.167642 1156 ts_tablet_manager.cc:1905] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.147258 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.147374 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 7cc18d1154ef4d05b3b82180a9c1ca8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.170040 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.147830 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.147928 744 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.170148 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a75d7986152f427d88ed34a443128bc6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.170210 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01d95356c7a7419bb56ddf67d451eadc" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.127194 1155 raft_consensus.cc:357] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.169173 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ec92eddfabf40918a6cabaff5fe113a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.170317 1155 raft_consensus.cc:383] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.169417 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12b85f220f3947ca8110f2db937e7b68" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.170367 1155 raft_consensus.cc:738] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.170450 771 raft_consensus.cc:2466] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.170498 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a792a26725954adea15ad57e31bfb224 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.169788 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b6d5ff92c343cb8f1b3039b84390af" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.170156 776 raft_consensus.cc:2466] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.170538 1155 consensus_queue.cc:260] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.170620 775 raft_consensus.cc:2466] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.168903 774 raft_consensus.cc:2466] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:03.170706 772 raft_consensus.cc:2466] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.171036 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bccdcd9de08648bb925db6fe56f97d20" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.171216 1155 ts_tablet_manager.cc:1428] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20250629 01:52:03.171425 1155 tablet_bootstrap.cc:492] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:03.144781 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 145534us
I20250629 01:52:03.180176 1155 tablet_bootstrap.cc:654] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.181411 986 leader_election.cc:304] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.181674 2758 raft_consensus.cc:2747] T d7e2c70221e94923b6a324a03f7133c6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.183777 986 leader_election.cc:343] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.183876 986 leader_election.cc:304] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.184059 2758 raft_consensus.cc:2747] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.185046 986 leader_election.cc:343] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.185144 986 leader_election.cc:304] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.185339 2758 raft_consensus.cc:2747] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.186197 986 leader_election.cc:304] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:03.186335 986 raft_consensus.cc:2684] T 97d0fbb691514d51a1ca71f32a666c2d P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:52:03.187458 1154 tablet_bootstrap.cc:492] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.187608 1154 ts_tablet_manager.cc:1397] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.002s
I20250629 01:52:03.188234 1154 raft_consensus.cc:357] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.188452 1154 raft_consensus.cc:383] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.188642 1154 raft_consensus.cc:738] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:52:03.188715 986 leader_election.cc:343] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.188807 986 leader_election.cc:304] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.188973 2758 raft_consensus.cc:2747] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.189198 1154 consensus_queue.cc:260] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.190515 1154 ts_tablet_manager.cc:1428] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250629 01:52:03.192514 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet c4530a43bdeb47d3ad29b17750be12d2
I20250629 01:52:03.193609 1154 tablet_bootstrap.cc:492] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.194047 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
/tmp/dist-test-task2ZKnTZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:35873
--webserver_interface=127.0.104.61
--webserver_port=43231
--builtin_ntp_servers=127.0.104.20:42941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34747,127.0.104.61:35873,127.0.104.60:32789
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250629 01:52:03.195039 1154 tablet_bootstrap.cc:654] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.196178 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet c4530a43bdeb47d3ad29b17750be12d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.197121 884 tablet_service.cc:1515] Processing DeleteTablet for tablet c4530a43bdeb47d3ad29b17750be12d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:03.202231 986 leader_election.cc:343] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.202329 986 leader_election.cc:304] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.202587 2758 raft_consensus.cc:2747] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.202979 986 leader_election.cc:304] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.203338 2758 raft_consensus.cc:2802] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.203401 2758 raft_consensus.cc:491] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.203485 2758 raft_consensus.cc:3058] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.205744 740 tablet_service.cc:1515] Processing DeleteTablet for tablet c4530a43bdeb47d3ad29b17750be12d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.210577 2758 raft_consensus.cc:513] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.210742 1155 tablet_bootstrap.cc:492] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.210840 1155 ts_tablet_manager.cc:1397] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.002s
I20250629 01:52:03.211477 2758 leader_election.cc:290] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.211611 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12b85f220f3947ca8110f2db937e7b68" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:03.212072 988 leader_election.cc:343] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.212582 986 leader_election.cc:304] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.212566 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_65aa0434a5d7401aa27aeaf83d5e1ecf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.213021 2758 raft_consensus.cc:2802] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.213085 2758 raft_consensus.cc:491] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250629 01:52:03.213088 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65aa0434a5d7401aa27aeaf83d5e1ecf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.213138 2758 raft_consensus.cc:3058] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.214201 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_122278fe170445189b568ab13655f5f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.214586 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_122278fe170445189b568ab13655f5f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.215214 2758 raft_consensus.cc:513] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.215471 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e50b86b187294b3fb9f068be98e6101d" } new_table_name: "default.Table_892aaaecf7a44b3982c3193e5d483997" modify_external_catalogs: true
I20250629 01:52:03.215946 2758 leader_election.cc:290] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.216416 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d0563326136c457aabee62f8d46d730c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.216446 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:03.216733 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0563326136c457aabee62f8d46d730c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:03.216864 988 leader_election.cc:343] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.217500 986 leader_election.cc:304] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.217850 2758 raft_consensus.cc:2802] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.217957 2758 raft_consensus.cc:491] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.217941 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_60a30fc239a64f36965d6eb696944710"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.218127 2758 raft_consensus.cc:3058] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.220202 2758 raft_consensus.cc:513] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.220929 1158 ts_tablet_manager.cc:1918] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250629 01:52:03.220985 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60a30fc239a64f36965d6eb696944710 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.221019 1158 log.cc:1199] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/43b845c3e7174618bd5c394d7206a634
I20250629 01:52:03.221117 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b6d5ff92c343cb8f1b3039b84390af" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.221261 1158 ts_tablet_manager.cc:1939] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:52:03.221567 988 leader_election.cc:343] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.221783 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 43b845c3e7174618bd5c394d7206a634 (table default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9 [id=ad6049db2ac64d5c86e06c58531e2120]) successfully deleted
I20250629 01:52:03.221969 2758 leader_election.cc:290] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.221920 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c65f50be17b4466a9991d2195fef4c5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.222265 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c65f50be17b4466a9991d2195fef4c5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.222337 986 leader_election.cc:304] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:03.222474 986 raft_consensus.cc:2684] T 43b845c3e7174618bd5c394d7206a634 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:52:03.223170 986 leader_election.cc:304] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.223521 2758 raft_consensus.cc:2802] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.223639 2758 raft_consensus.cc:491] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.223768 2758 raft_consensus.cc:3058] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.151726 2859 log.cc:1199] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/819ed44d98f14075b9a67b88dc44e86e
I20250629 01:52:03.224251 2859 ts_tablet_manager.cc:1939] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:03.224463 2845 raft_consensus.cc:491] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.224694 2845 raft_consensus.cc:513] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.225205 2845 leader_election.cc:290] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:03.225760 850 leader_election.cc:343] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 45ca747f03524076b4112e4db78614fe
I20250629 01:52:03.226820 2859 tablet_replica.cc:331] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:03.227186 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 819ed44d98f14075b9a67b88dc44e86e (table default.Table_f6acf9f470204fdb876176dc5c191895 [id=0070a00fcaa945cfb2e47f3c3e347323]) successfully deleted
I20250629 01:52:03.227792 2859 raft_consensus.cc:2241] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.227928 2859 raft_consensus.cc:2270] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.228052 449 master_service.cc:948] ReplaceTablet: received request to replace tablet 28da32ab17024f519105c2490efd9136 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:03.228402 2859 ts_tablet_manager.cc:1905] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.229876 1155 raft_consensus.cc:357] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.230055 1155 raft_consensus.cc:383] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.230105 1155 raft_consensus.cc:738] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.230280 1155 consensus_queue.cc:260] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.232692 1154 tablet_bootstrap.cc:492] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.232800 1154 ts_tablet_manager.cc:1397] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
W20250629 01:52:03.232954 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 1ac60a024a074953b513ae4f60840d25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac60a024a074953b513ae4f60840d25 already in progress: creating tablet
W20250629 01:52:03.233114 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 12873ec9a5574e789f9d669ddda85704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12873ec9a5574e789f9d669ddda85704 already in progress: creating tablet
I20250629 01:52:03.233346 1154 raft_consensus.cc:357] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.233466 1154 raft_consensus.cc:383] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.233502 1154 raft_consensus.cc:738] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.233640 1154 consensus_queue.cc:260] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.234229 1154 ts_tablet_manager.cc:1428] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:03.234431 1154 tablet_bootstrap.cc:492] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.234449 1159 tablet_bootstrap.cc:492] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.234560 1159 ts_tablet_manager.cc:1397] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.005s
I20250629 01:52:03.235368 1159 raft_consensus.cc:357] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.235512 1159 raft_consensus.cc:383] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.235564 1159 raft_consensus.cc:738] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.235769 1159 consensus_queue.cc:260] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:03.236146 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4416bbec7d1c40df9a835b503d368d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4416bbec7d1c40df9a835b503d368d4c already in progress: creating tablet
W20250629 01:52:03.237188 986 leader_election.cc:343] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.237270 986 leader_election.cc:304] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.237610 2834 raft_consensus.cc:2747] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.237946 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83e0f20f27264ff5b064f14638029f79 consensus state has no leader
I20250629 01:52:03.238965 1155 ts_tablet_manager.cc:1428] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20250629 01:52:03.239493 2651 consensus_queue.cc:1035] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:03.239953 2794 tablet.cc:1722] T 9c6239dd09a343d59da51d114f48658a P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:03.242564 1155 tablet_bootstrap.cc:492] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:03.243512 850 leader_election.cc:343] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 819ed44d98f14075b9a67b88dc44e86e
I20250629 01:52:03.243599 850 leader_election.cc:304] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:03.243752 850 raft_consensus.cc:2684] T 819ed44d98f14075b9a67b88dc44e86e P 443c4bd3c5d548ee8cd347d2b4de5293: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:52:03.244467 2852 tablet.cc:1722] T 9c6239dd09a343d59da51d114f48658a P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:03.246340 1156 ts_tablet_manager.cc:1918] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:03.246418 1156 log.cc:1199] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/e5d27d658892496a9b19b57949a8399a
I20250629 01:52:03.246680 1156 ts_tablet_manager.cc:1939] T e5d27d658892496a9b19b57949a8399a P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.247241 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1392) took 1709 ms. Trace:
I20250629 01:52:03.247347 1159 ts_tablet_manager.cc:1428] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250629 01:52:03.247311 1156 rpcz_store.cc:276] 0629 01:52:01.537719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.537841 (+ 122us) service_pool.cc:225] Handling call
0629 01:52:03.162871 (+1625030us) ts_tablet_manager.cc:1145] Deleting tablet e5d27d658892496a9b19b57949a8399a
0629 01:52:03.162883 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.188491 (+ 25608us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.246274 (+ 57783us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.247219 (+ 945us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1624871}
I20250629 01:52:03.247552 1156 tablet_replica.cc:331] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:03.247678 1156 raft_consensus.cc:2241] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:03.247869 1156 pending_rounds.cc:70] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:03.247900 1156 pending_rounds.cc:77] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759230958600192 op_type: NO_OP noop_request { }
I20250629 01:52:03.247958 1156 raft_consensus.cc:2887] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:03.248009 1156 raft_consensus.cc:2270] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.248317 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet e5d27d658892496a9b19b57949a8399a (table default.Table_768dd2c93bfa4a828037ed23b11dc5ae [id=e8832475ef554a2ebbf61f3481bf197c]) successfully deleted
I20250629 01:52:03.248407 1156 ts_tablet_manager.cc:1905] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.250589 1159 tablet_bootstrap.cc:492] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.252252 1155 tablet_bootstrap.cc:654] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.252324 1159 tablet_bootstrap.cc:654] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:03.255029 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 13d779335d09442682554322b59f915a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13d779335d09442682554322b59f915a already in progress: creating tablet
I20250629 01:52:03.260365 2758 raft_consensus.cc:513] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:03.261049 712 consensus_peers.cc:489] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.261119 454 catalog_manager.cc:5582] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.261478 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
W20250629 01:52:03.261649 712 consensus_peers.cc:489] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.261648 454 catalog_manager.cc:5582] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.262223 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:03.262403 454 catalog_manager.cc:5582] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.262723 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet b1ebbcb2c26b4667b2405154a9d8fccb (table default.Table_a2726ab5a15a4cf8833868289c7ca5ee [id=70f6cbc3d10848f9af1a4c482c27eeb8]). Expected version 1 got 0
I20250629 01:52:03.262893 454 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d6b6d5ff92c343cb8f1b3039b84390af (table default.Table_9c12a09abc39412497235b71efb0ea8e [id=8038bbd131674ed59dbe36c73c14d466]). Expected version 3 got 0
I20250629 01:52:03.263533 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7b3fdda8530a4fefbb38859d44f183af" } modify_external_catalogs: true
I20250629 01:52:03.263650 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7b3fdda8530a4fefbb38859d44f183af" } modify_external_catalogs: true
I20250629 01:52:03.263840 1030 tablet_service.cc:1468] Processing CreateTablet for tablet efaffb348f5d486a9735e1f41351d702 (DEFAULT_TABLE table=default.Table_a65d27fa1e0a4e18b5133bb7a8ccd201 [id=831f234fdd224261aa968700bf3ad289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.264168 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efaffb348f5d486a9735e1f41351d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.265995 2548 raft_consensus.cc:491] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.266091 2548 raft_consensus.cc:513] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.266534 2548 leader_election.cc:290] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.267081 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_540c577da2d044a19efbc470511fcf47" } modify_external_catalogs: true
I20250629 01:52:03.267185 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_540c577da2d044a19efbc470511fcf47" } modify_external_catalogs: true
I20250629 01:52:03.267318 2758 leader_election.cc:290] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:03.267753 712 consensus_peers.cc:489] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.268289 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 0af8f23e99b34612bf7a65a2f0caf2a4 (DEFAULT_TABLE table=default.Table_f3b844a43dbc4e268724542a0d465f0b [id=24446b36f3064421b8e7e08cfa492527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.268564 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af8f23e99b34612bf7a65a2f0caf2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.271823 744 tablet_service.cc:1468] Processing CreateTablet for tablet 0af8f23e99b34612bf7a65a2f0caf2a4 (DEFAULT_TABLE table=default.Table_f3b844a43dbc4e268724542a0d465f0b [id=24446b36f3064421b8e7e08cfa492527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.272132 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af8f23e99b34612bf7a65a2f0caf2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.272213 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "12b85f220f3947ca8110f2db937e7b68" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.272352 774 raft_consensus.cc:3058] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.272799 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b6d5ff92c343cb8f1b3039b84390af" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.272908 776 raft_consensus.cc:3058] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.274685 774 raft_consensus.cc:2466] T 12b85f220f3947ca8110f2db937e7b68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:03.274886 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.275008 775 raft_consensus.cc:3058] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.275086 2534 raft_consensus.cc:491] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.275162 2534 raft_consensus.cc:513] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.275557 2534 leader_election.cc:290] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.275008 776 raft_consensus.cc:2466] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:03.276368 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a75d7986152f427d88ed34a443128bc6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.276387 740 tablet_service.cc:1468] Processing CreateTablet for tablet efaffb348f5d486a9735e1f41351d702 (DEFAULT_TABLE table=default.Table_a65d27fa1e0a4e18b5133bb7a8ccd201 [id=831f234fdd224261aa968700bf3ad289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.276489 776 raft_consensus.cc:3058] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.276722 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efaffb348f5d486a9735e1f41351d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.277164 775 raft_consensus.cc:2466] T c62678645ff54aa6b3809212f4d66ec2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:03.278558 776 raft_consensus.cc:2466] T a75d7986152f427d88ed34a443128bc6 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
W20250629 01:52:03.279467 710 leader_election.cc:343] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 7cc18d1154ef4d05b3b82180a9c1ca8b
I20250629 01:52:03.280334 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4f0e6d014e394eb7893f4227a0e8546a (DEFAULT_TABLE table=default.Table_222ce79cd46a42eea3f0354709d057ea [id=3edf20ef00564d6c809dcd756f33b784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.281116 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f0e6d014e394eb7893f4227a0e8546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.282287 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ec92eddfabf40918a6cabaff5fe113a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:03.282428 1052 raft_consensus.cc:2466] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:03.285744 986 leader_election.cc:304] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.287297 710 leader_election.cc:304] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:03.287530 2837 raft_consensus.cc:2802] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.287593 2837 raft_consensus.cc:491] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.287643 2837 raft_consensus.cc:3058] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.287707 2834 raft_consensus.cc:2802] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.287798 2834 raft_consensus.cc:695] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:03.288012 2834 consensus_queue.cc:237] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.288914 1154 tablet_bootstrap.cc:654] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.289841 2837 raft_consensus.cc:513] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.290293 2837 leader_election.cc:290] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.290508 986 leader_election.cc:304] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.291002 2534 raft_consensus.cc:491] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.291111 2764 raft_consensus.cc:2802] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.291091 2534 raft_consensus.cc:513] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.291241 2764 raft_consensus.cc:695] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:03.291427 2764 consensus_queue.cc:237] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.291532 2534 leader_election.cc:290] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.291785 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ec92eddfabf40918a6cabaff5fe113a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:03.291934 1052 raft_consensus.cc:3058] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.292995 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4f0e6d014e394eb7893f4227a0e8546a (DEFAULT_TABLE table=default.Table_222ce79cd46a42eea3f0354709d057ea [id=3edf20ef00564d6c809dcd756f33b784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.293251 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f0e6d014e394eb7893f4227a0e8546a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:03.294520 710 leader_election.cc:343] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ec0ca04e5f6b42df8567deb53a3e5f53
I20250629 01:52:03.295588 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 08dd5ec4c05f4408a3b832161199140b (DEFAULT_TABLE table=default.Table_702cf8b71dad4ba9a893b84bf903252b [id=102bfa02489d46738e10d4916786d347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.295928 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08dd5ec4c05f4408a3b832161199140b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.296344 750 tablet_service.cc:1468] Processing CreateTablet for tablet 08dd5ec4c05f4408a3b832161199140b (DEFAULT_TABLE table=default.Table_702cf8b71dad4ba9a893b84bf903252b [id=102bfa02489d46738e10d4916786d347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.296689 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08dd5ec4c05f4408a3b832161199140b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.301445 1052 raft_consensus.cc:2466] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:03.302662 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 4c07a1580c734aeba695151d23c7e020 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c07a1580c734aeba695151d23c7e020 already in progress: creating tablet
W20250629 01:52:03.303217 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 790a1347e57f4cea895132564eafae9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 790a1347e57f4cea895132564eafae9c already in progress: creating tablet
W20250629 01:52:03.303930 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet c4530a43bdeb47d3ad29b17750be12d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4530a43bdeb47d3ad29b17750be12d2 already in progress: creating tablet
I20250629 01:52:03.307343 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:03.307652 452 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet b15300fd8ca7489f9d77f1f8992f8df3 (table default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da]). Expected version 1 got 0
I20250629 01:52:03.313513 751 tablet_service.cc:1468] Processing CreateTablet for tablet 28bc247520074b70973312e183ed6796 (DEFAULT_TABLE table=default.Table_78bb87e44b9e4ceea207030a95646888 [id=6c973b28cf4a412e8e9a6227fc9f53f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.313412 455 catalog_manager.cc:5582] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.313789 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28bc247520074b70973312e183ed6796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.313905 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:03.314749 455 catalog_manager.cc:5582] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.315405 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:03.315829 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 9c6239dd09a343d59da51d114f48658a (table default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7]). Expected version 2 got 0
I20250629 01:52:03.316017 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d6b6d5ff92c343cb8f1b3039b84390af (table default.Table_9c12a09abc39412497235b71efb0ea8e [id=8038bbd131674ed59dbe36c73c14d466]). Expected version 3 got 0
W20250629 01:52:03.257678 850 leader_election.cc:343] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 3a81e6f9711c4d398ae09f33a75566ae
I20250629 01:52:03.319424 850 leader_election.cc:304] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:03.319758 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 62083.5us
I20250629 01:52:03.270882 915 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ec92eddfabf40918a6cabaff5fe113a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.320325 2845 raft_consensus.cc:2747] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.320461 2845 raft_consensus.cc:491] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.320567 2845 raft_consensus.cc:513] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.321120 2845 leader_election.cc:290] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.321324 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a75d7986152f427d88ed34a443128bc6" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.271569 2855 tablet.cc:1722] T 9c6239dd09a343d59da51d114f48658a P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:03.290179 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cc18d1154ef4d05b3b82180a9c1ca8b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.306617 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ec92eddfabf40918a6cabaff5fe113a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.315236 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec0ca04e5f6b42df8567deb53a3e5f53" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.322443 710 consensus_peers.cc:489] T 47cbc958eca14ed69fb45c3c2984aeef P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47cbc958eca14ed69fb45c3c2984aeef. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.322948 455 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:03.301927 986 leader_election.cc:304] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.324541 710 leader_election.cc:304] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:03.324579 986 leader_election.cc:304] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:52:03.324836 850 leader_election.cc:343] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 9fe9674ffeb44ba6ab60de59d4c08670
I20250629 01:52:03.324911 879 tablet_service.cc:1468] Processing CreateTablet for tablet 0af8f23e99b34612bf7a65a2f0caf2a4 (DEFAULT_TABLE table=default.Table_f3b844a43dbc4e268724542a0d465f0b [id=24446b36f3064421b8e7e08cfa492527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.324997 2760 raft_consensus.cc:2802] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.325124 884 tablet_service.cc:1468] Processing CreateTablet for tablet efaffb348f5d486a9735e1f41351d702 (DEFAULT_TABLE table=default.Table_a65d27fa1e0a4e18b5133bb7a8ccd201 [id=831f234fdd224261aa968700bf3ad289]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.325197 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af8f23e99b34612bf7a65a2f0caf2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.325390 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4f0e6d014e394eb7893f4227a0e8546a (DEFAULT_TABLE table=default.Table_222ce79cd46a42eea3f0354709d057ea [id=3edf20ef00564d6c809dcd756f33b784]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.325656 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f0e6d014e394eb7893f4227a0e8546a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.304694 1154 tablet_bootstrap.cc:492] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.325879 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efaffb348f5d486a9735e1f41351d702. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.325994 1154 ts_tablet_manager.cc:1397] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.001s
I20250629 01:52:03.326000 892 tablet_service.cc:1468] Processing CreateTablet for tablet 08dd5ec4c05f4408a3b832161199140b (DEFAULT_TABLE table=default.Table_702cf8b71dad4ba9a893b84bf903252b [id=102bfa02489d46738e10d4916786d347]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.326262 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08dd5ec4c05f4408a3b832161199140b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.326579 1154 raft_consensus.cc:357] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.326730 1154 raft_consensus.cc:383] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.326794 1154 raft_consensus.cc:738] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.327927 1154 consensus_queue.cc:260] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.324919 850 leader_election.cc:304] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:03.333170 850 leader_election.cc:343] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: f94d8fc1b6254ba69acadcaff3613d97
I20250629 01:52:03.333242 850 leader_election.cc:304] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:03.333482 850 leader_election.cc:343] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:52:03.333567 850 leader_election.cc:304] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.333788 2670 raft_consensus.cc:2747] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.333887 2670 raft_consensus.cc:2747] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.333990 2670 raft_consensus.cc:2747] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.336225 2534 raft_consensus.cc:2802] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.336341 2534 raft_consensus.cc:695] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:03.336544 2534 consensus_queue.cc:237] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.346854 2758 raft_consensus.cc:2802] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
W20250629 01:52:03.360864 712 leader_election.cc:343] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.361115 712 leader_election.cc:343] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.361195 712 leader_election.cc:304] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
W20250629 01:52:03.361487 712 leader_election.cc:343] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.361583 2651 raft_consensus.cc:2747] T 7cc18d1154ef4d05b3b82180a9c1ca8b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.361688 712 leader_election.cc:343] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.361761 712 leader_election.cc:304] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:03.362887 2834 raft_consensus.cc:491] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.363191 2834 raft_consensus.cc:513] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.363806 2834 leader_election.cc:290] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.368170 2651 raft_consensus.cc:2747] T ec0ca04e5f6b42df8567deb53a3e5f53 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.377509 1155 tablet_bootstrap.cc:492] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.377652 1155 ts_tablet_manager.cc:1397] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.135s user 0.001s sys 0.005s
I20250629 01:52:03.378238 1155 raft_consensus.cc:357] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.378407 1155 raft_consensus.cc:383] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.378491 1155 raft_consensus.cc:738] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.378706 1155 consensus_queue.cc:260] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.379325 1155 ts_tablet_manager.cc:1428] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:03.380411 1155 tablet_bootstrap.cc:492] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.381896 1155 tablet_bootstrap.cc:654] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.382643 2845 raft_consensus.cc:491] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.382745 2845 raft_consensus.cc:513] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.383152 2845 leader_election.cc:290] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.385067 1156 ts_tablet_manager.cc:1918] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:03.386020 1156 log.cc:1199] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/ecbfc4e45dc9435bbd7a55ba7b7e4155
I20250629 01:52:03.386384 1156 ts_tablet_manager.cc:1939] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.387003 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1390) took 1849 ms. Trace:
I20250629 01:52:03.388418 2760 raft_consensus.cc:695] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
W20250629 01:52:03.388477 850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 54990.1us
I20250629 01:52:03.388540 1156 rpcz_store.cc:276] 0629 01:52:01.537544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.537962 (+ 418us) service_pool.cc:225] Handling call
0629 01:52:03.247521 (+1709559us) ts_tablet_manager.cc:1145] Deleting tablet ecbfc4e45dc9435bbd7a55ba7b7e4155
0629 01:52:03.247528 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.250706 (+ 3178us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.384993 (+134287us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.386989 (+ 1996us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709419}
W20250629 01:52:03.388677 850 leader_election.cc:343] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 45ca747f03524076b4112e4db78614fe
I20250629 01:52:03.388760 850 leader_election.cc:304] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.388757 2760 consensus_queue.cc:237] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.389050 1156 tablet_replica.cc:331] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
W20250629 01:52:03.389268 988 leader_election.cc:343] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.389528 1156 raft_consensus.cc:2241] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:03.389775 1156 pending_rounds.cc:70] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:03.389845 1156 pending_rounds.cc:77] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759229577297920 op_type: NO_OP noop_request { }
I20250629 01:52:03.389962 1156 raft_consensus.cc:2887] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:03.390764 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f465d0bb46674763abfc8d93f45cd13d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.390947 879 tablet_service.cc:1468] Processing CreateTablet for tablet 28bc247520074b70973312e183ed6796 (DEFAULT_TABLE table=default.Table_78bb87e44b9e4ceea207030a95646888 [id=6c973b28cf4a412e8e9a6227fc9f53f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.391254 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28bc247520074b70973312e183ed6796. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:03.391396 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 (table default.Table_fed4bed522aa4c8192e79303877c5472 [id=ada9bbf9d1b74ede92614313092c90dc]) successfully deleted
I20250629 01:52:03.392625 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 28bc247520074b70973312e183ed6796 (DEFAULT_TABLE table=default.Table_78bb87e44b9e4ceea207030a95646888 [id=6c973b28cf4a412e8e9a6227fc9f53f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:03.392941 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28bc247520074b70973312e183ed6796. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:03.387998 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 63092.5us
I20250629 01:52:03.395804 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:03.397217 2896 raft_consensus.cc:2747] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.397981 1159 tablet_bootstrap.cc:492] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.398093 1159 ts_tablet_manager.cc:1397] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.000s
I20250629 01:52:03.399102 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_79af1ae8dbb242a9a55e80e72378d993" } modify_external_catalogs: true
I20250629 01:52:03.399262 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_79af1ae8dbb242a9a55e80e72378d993" } modify_external_catalogs: true
I20250629 01:52:03.399581 1159 raft_consensus.cc:357] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.399724 1159 raft_consensus.cc:383] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.399775 1159 raft_consensus.cc:738] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.399966 1159 consensus_queue.cc:260] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.400175 1156 raft_consensus.cc:2270] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.400394 1159 ts_tablet_manager.cc:1428] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:03.400403 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5ef79ed619d54eb28e78246a1a4f2b0f" } new_table_name: "default.Table_e8bfbf3944304871b5519a4a93d9590e" modify_external_catalogs: true
I20250629 01:52:03.400928 1156 ts_tablet_manager.cc:1905] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.401456 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6428b6c04c7b4884b29cdc13ccf39be2" } new_table_name: "default.Table_d5d4609ee3c344c19f141a737ae4e9b7" modify_external_catalogs: true
I20250629 01:52:03.391115 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a12fffe61d7140308acd181bfa3d2700" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.403405 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5a50c642fa1422bb612d0edeadb4ee7" } new_table_name: "default.Table_8e919f677ab449b2af29d551b3d7cb58" modify_external_catalogs: true
I20250629 01:52:03.404464 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 462789e8daef42a08ff8b2cd2ddd1c49
I20250629 01:52:03.404425 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_70b23bddafa9412880fb0cb669f1c6d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.404816 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70b23bddafa9412880fb0cb669f1c6d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.405042 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_444d56c9689c4f32b190cbde3ab315af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.405385 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_444d56c9689c4f32b190cbde3ab315af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.398690 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f465d0bb46674763abfc8d93f45cd13d" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.406524 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 462789e8daef42a08ff8b2cd2ddd1c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.407085 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 462789e8daef42a08ff8b2cd2ddd1c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.407747 1155 tablet_bootstrap.cc:492] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.407833 1155 ts_tablet_manager.cc:1397] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20250629 01:52:03.408675 1155 raft_consensus.cc:357] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.408854 1155 raft_consensus.cc:383] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.408931 1155 raft_consensus.cc:738] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.409133 1155 consensus_queue.cc:260] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.409778 1155 ts_tablet_manager.cc:1428] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:03.410015 1155 tablet_bootstrap.cc:492] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.411692 1155 tablet_bootstrap.cc:654] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.413512 1154 ts_tablet_manager.cc:1428] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.087s user 0.006s sys 0.002s
I20250629 01:52:03.414096 1154 tablet_bootstrap.cc:492] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.415366 1154 tablet_bootstrap.cc:654] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.416182 1158 tablet_replica.cc:331] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.416338 1158 raft_consensus.cc:2241] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.416478 1158 raft_consensus.cc:2270] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.416905 1158 ts_tablet_manager.cc:1905] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.417037 2758 raft_consensus.cc:695] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:03.417265 2758 consensus_queue.cc:237] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:03.418046 988 leader_election.cc:343] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.418766 850 leader_election.cc:343] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 1fe0cd4b8f7d47268a80f645bc8b2b9d
I20250629 01:52:03.419373 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 98921f32d78c4b1ca80576fc805c466d
I20250629 01:52:03.419656 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_65704a9f46f54a6d8ac7cd55be389d28" } new_table_name: "default.Table_aa8f14fe50764434b6dfb679cf7a0c71" modify_external_catalogs: true
I20250629 01:52:03.419857 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 10920d584d0c456d8a5eb8efb81da36f
W20250629 01:52:03.419951 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e119711a47f14100a9455a1afd84b6d1 consensus state has no leader
I20250629 01:52:03.420189 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bb9d3aa0a3084c80a33203f45b4978eb" } modify_external_catalogs: true
I20250629 01:52:03.420281 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_bb9d3aa0a3084c80a33203f45b4978eb" } modify_external_catalogs: true
I20250629 01:52:03.420400 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f14a24eca5f842b19f4361d7bf57557a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.420773 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f14a24eca5f842b19f4361d7bf57557a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:03.421190 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5064e7b3d4d14f868e9aba9c3175382d consensus state has no leader
I20250629 01:52:03.421433 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0070d2f697c64e168b71411e800f6823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.421783 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0070d2f697c64e168b71411e800f6823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.423357 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_0a60f2cb55074b998ddf4c8a1c30ea84 [id=2ebdb5f48c7b417bb521dd2271922d56] alter complete (version 1)
I20250629 01:52:03.424952 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_226a81ddbfc54450b599ac93e64a0517" } modify_external_catalogs: true
I20250629 01:52:03.425056 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_226a81ddbfc54450b599ac93e64a0517" } modify_external_catalogs: true
I20250629 01:52:03.425885 1159 tablet_bootstrap.cc:492] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:03.426096 986 leader_election.cc:343] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.426172 986 leader_election.cc:304] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.426738 2898 raft_consensus.cc:2747] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.426831 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c5f2016e1a584e959460454379b34a60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.426993 1159 tablet_bootstrap.cc:654] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:03.427150 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5f2016e1a584e959460454379b34a60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.434463 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_30f3d1b749884208a35c3f0a442b075b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.434836 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30f3d1b749884208a35c3f0a442b075b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.436139 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 10920d584d0c456d8a5eb8efb81da36f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.436627 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ec3937679b37476f934aceb30ac62d68 with delete_type TABLET_DATA_DELETED (replaced by tablet 966e379adef8435887a2d8f39bb99968) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.437058 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet bb98aa9d365541969ea1dc70960760b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.437507 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe1f8083a884cdd97afce83bd22073c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.437892 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.438243 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.438596 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ef3060d46ac24330be49103920dc1dee with delete_type TABLET_DATA_DELETED (replaced by tablet 86099eb9e40a4ea389afc059f90a97ef) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.438964 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 12b85f220f3947ca8110f2db937e7b68 with delete_type TABLET_DATA_DELETED (replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.439385 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.439746 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 01d95356c7a7419bb56ddf67d451eadc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.440119 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a75d7986152f427d88ed34a443128bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.440483 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet c5d45854191247bf8172d0bbfd01b6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.441031 740 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.441395 740 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.441383 2896 raft_consensus.cc:491] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.441489 2896 raft_consensus.cc:513] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.441780 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a75d7986152f427d88ed34a443128bc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.442289 2896 leader_election.cc:290] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.442488 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 10920d584d0c456d8a5eb8efb81da36f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.442689 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 12b85f220f3947ca8110f2db937e7b68 with delete_type TABLET_DATA_DELETED (replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca) from {username='slave'} at 127.0.0.1:38530
W20250629 01:52:03.445966 850 leader_election.cc:343] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 1fe0cd4b8f7d47268a80f645bc8b2b9d
I20250629 01:52:03.446051 850 leader_election.cc:304] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.446347 850 leader_election.cc:304] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.446535 2845 raft_consensus.cc:2747] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.446656 2845 raft_consensus.cc:2747] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.446796 2794 tablet.cc:1722] T a12fffe61d7140308acd181bfa3d2700 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:52:03.447146 710 consensus_peers.cc:489] T 462789e8daef42a08ff8b2cd2ddd1c49 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.447988 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 10920d584d0c456d8a5eb8efb81da36f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.448251 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 327c303b6b9347f5b4cb78f74d41a44a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.448518 884 tablet_service.cc:1515] Processing DeleteTablet for tablet a014907d09c94745859436f1eb73b280 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.448508 2852 tablet.cc:1722] T a12fffe61d7140308acd181bfa3d2700 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:03.449558 2855 tablet.cc:1722] T 3d87639255ec46d98fcfb0f9638ee39a P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:03.450033 2794 tablet.cc:1722] T 3d87639255ec46d98fcfb0f9638ee39a P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:52:03.453204 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72f277d4234744a2995953a5eb1715b3 consensus state has no leader
I20250629 01:52:03.453578 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e258174bcbf9402daa2cb5b1127139b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.453943 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e258174bcbf9402daa2cb5b1127139b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.454169 2548 consensus_queue.cc:1035] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:03.458334 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_73fffb9e284e4a57a7a64fb947efa644" } new_table_name: "default.Table_e1476c42392a4be5bb19d9c0879e2d8f" modify_external_catalogs: true
I20250629 01:52:03.462903 1156 ts_tablet_manager.cc:1918] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:03.464196 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet 3d87639255ec46d98fcfb0f9638ee39a (table default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0]). Expected version 2 got 0
I20250629 01:52:03.464448 451 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b1ebbcb2c26b4667b2405154a9d8fccb (table default.Table_a2726ab5a15a4cf8833868289c7ca5ee [id=70f6cbc3d10848f9af1a4c482c27eeb8]). Expected version 1 got 0
I20250629 01:52:03.466051 1156 log.cc:1199] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/5ab0aa4e74d345b2a214b90b5018c82b
I20250629 01:52:03.466352 1156 ts_tablet_manager.cc:1939] T 5ab0aa4e74d345b2a214b90b5018c82b P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.466989 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1393) took 1929 ms. Trace:
I20250629 01:52:03.467064 1156 rpcz_store.cc:276] 0629 01:52:01.537790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.538022 (+ 232us) service_pool.cc:225] Handling call
0629 01:52:03.389014 (+1850992us) ts_tablet_manager.cc:1145] Deleting tablet 5ab0aa4e74d345b2a214b90b5018c82b
0629 01:52:03.389021 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.450282 (+ 61261us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.462836 (+ 12554us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.466973 (+ 4137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1850867}
I20250629 01:52:03.467356 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1394) took 1929 ms. Trace:
I20250629 01:52:03.467413 1156 rpcz_store.cc:276] 0629 01:52:01.537919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.538123 (+ 204us) service_pool.cc:225] Handling call
0629 01:52:03.467267 (+1929144us) ts_tablet_manager.cc:1145] Deleting tablet ee7fb74854db4034a9d44f6d025e6756
0629 01:52:03.467274 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.467345 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929062}
I20250629 01:52:03.467646 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1400) took 1680 ms. Trace:
I20250629 01:52:03.467700 1156 rpcz_store.cc:276] 0629 01:52:01.787054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.787108 (+ 54us) service_pool.cc:225] Handling call
0629 01:52:03.467578 (+1680470us) ts_tablet_manager.cc:1145] Deleting tablet 4ee1e7985f4b4422ba607962bf1fdc42
0629 01:52:03.467584 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.467636 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1680332}
I20250629 01:52:03.464195 1155 tablet_bootstrap.cc:492] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.468245 1155 ts_tablet_manager.cc:1397] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20250629 01:52:03.468634 1158 ts_tablet_manager.cc:1918] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:03.468729 1158 log.cc:1199] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/462789e8daef42a08ff8b2cd2ddd1c49
I20250629 01:52:03.468814 1155 raft_consensus.cc:357] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.468941 1155 raft_consensus.cc:383] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.468981 1155 raft_consensus.cc:738] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.469050 1158 ts_tablet_manager.cc:1939] T 462789e8daef42a08ff8b2cd2ddd1c49 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.469143 1155 consensus_queue.cc:260] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.469513 1155 ts_tablet_manager.cc:1428] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:03.469725 1155 tablet_bootstrap.cc:492] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.471001 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 462789e8daef42a08ff8b2cd2ddd1c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.471087 1155 tablet_bootstrap.cc:654] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.480537 879 tablet_service.cc:1515] Processing DeleteTablet for tablet e4d6f1eb43b24fda94aa0afcf4501195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.482466 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1412) took 1612 ms. Trace:
I20250629 01:52:03.482604 1156 rpcz_store.cc:276] 0629 01:52:01.855819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.855867 (+ 48us) service_pool.cc:225] Handling call
0629 01:52:03.467841 (+1611974us) ts_tablet_manager.cc:1145] Deleting tablet 1d2e2e2a1df5473392d9bc2d2ca9a993
0629 01:52:03.467847 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.467896 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611846}
I20250629 01:52:03.482976 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1414) took 1627 ms. Trace:
I20250629 01:52:03.483083 1156 rpcz_store.cc:276] 0629 01:52:01.855949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.856127 (+ 178us) service_pool.cc:225] Handling call
0629 01:52:03.482877 (+1626750us) ts_tablet_manager.cc:1145] Deleting tablet 0c422ceed7594a18b32e5f70350e5b7b
0629 01:52:03.482886 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.482962 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1626657}
I20250629 01:52:03.483485 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1413) took 1627 ms. Trace:
I20250629 01:52:03.483592 1156 rpcz_store.cc:276] 0629 01:52:01.855891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.856025 (+ 134us) service_pool.cc:225] Handling call
0629 01:52:03.483384 (+1627359us) ts_tablet_manager.cc:1145] Deleting tablet bb98aa9d365541969ea1dc70960760b2
0629 01:52:03.483392 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.483459 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627139}
I20250629 01:52:03.483924 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1415) took 1627 ms. Trace:
I20250629 01:52:03.484028 1156 rpcz_store.cc:276] 0629 01:52:01.856014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.856256 (+ 242us) service_pool.cc:225] Handling call
0629 01:52:03.483840 (+1627584us) ts_tablet_manager.cc:1145] Deleting tablet 78e379d748434664b4e2684d3f6012ab
0629 01:52:03.483848 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.483911 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1627488}
I20250629 01:52:03.484372 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1411) took 1628 ms. Trace:
I20250629 01:52:03.484485 1156 rpcz_store.cc:276] 0629 01:52:01.855728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.931914 (+ 76186us) service_pool.cc:225] Handling call
0629 01:52:03.484293 (+1552379us) ts_tablet_manager.cc:1145] Deleting tablet e45d1c8068e346039524312a058e9de7
0629 01:52:03.484300 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.484358 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1552240}
W20250629 01:52:03.484596 988 consensus_peers.cc:489] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:03.486970 850 leader_election.cc:343] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: a014907d09c94745859436f1eb73b280
I20250629 01:52:03.490180 1158 tablet_replica.cc:331] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.490233 1156 tablet_replica.cc:331] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:03.490356 1158 raft_consensus.cc:2241] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.490401 1156 raft_consensus.cc:2241] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.490470 1158 raft_consensus.cc:2270] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.490523 1156 raft_consensus.cc:2270] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.490901 1158 ts_tablet_manager.cc:1905] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.490936 1156 ts_tablet_manager.cc:1905] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:03.485852 712 consensus_peers.cc:489] T d062fb830020442faee626be006926c1 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.500028 1159 tablet_bootstrap.cc:492] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
W20250629 01:52:03.500283 850 leader_election.cc:343] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: a014907d09c94745859436f1eb73b280
I20250629 01:52:03.500519 850 leader_election.cc:304] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.500578 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:52:03.500619 1159 ts_tablet_manager.cc:1397] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.000s
I20250629 01:52:03.501240 1159 raft_consensus.cc:357] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.504320 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 5ab0aa4e74d345b2a214b90b5018c82b (table default.Table_7253ebd64bc14d1c89a2602b5f64e13a [id=e5609a082dbe41fb9e7fa7dc3d717297]) successfully deleted
W20250629 01:52:03.504536 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet ee7fb74854db4034a9d44f6d025e6756 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee7fb74854db4034a9d44f6d025e6756 already in progress: creating tablet
W20250629 01:52:03.504709 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 4ee1e7985f4b4422ba607962bf1fdc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ee1e7985f4b4422ba607962bf1fdc42 already in progress: creating tablet
W20250629 01:52:03.504837 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d2e2e2a1df5473392d9bc2d2ca9a993 already in progress: creating tablet
W20250629 01:52:03.504961 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 0c422ceed7594a18b32e5f70350e5b7b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c422ceed7594a18b32e5f70350e5b7b already in progress: creating tablet
W20250629 01:52:03.505072 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet bb98aa9d365541969ea1dc70960760b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb98aa9d365541969ea1dc70960760b2 already in progress: creating tablet
W20250629 01:52:03.505206 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet 78e379d748434664b4e2684d3f6012ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e379d748434664b4e2684d3f6012ab already in progress: creating tablet
W20250629 01:52:03.505312 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet e45d1c8068e346039524312a058e9de7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e45d1c8068e346039524312a058e9de7 already in progress: creating tablet
I20250629 01:52:03.510037 1158 ts_tablet_manager.cc:1918] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.510131 1158 log.cc:1199] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ec3937679b37476f934aceb30ac62d68
I20250629 01:52:03.510352 1158 ts_tablet_manager.cc:1939] T ec3937679b37476f934aceb30ac62d68 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.504551 2896 raft_consensus.cc:2747] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.514051 2859 ts_tablet_manager.cc:1918] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.516631 2859 log.cc:1199] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/daf312e60c014fe698d3ec9f1a1b769c
I20250629 01:52:03.517148 2859 ts_tablet_manager.cc:1939] T daf312e60c014fe698d3ec9f1a1b769c P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:03.517349 1159 raft_consensus.cc:383] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.517445 1159 raft_consensus.cc:738] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.517637 1154 tablet_bootstrap.cc:492] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.517728 2896 raft_consensus.cc:491] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.517792 1154 ts_tablet_manager.cc:1397] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.104s user 0.022s sys 0.004s
I20250629 01:52:03.517863 2896 raft_consensus.cc:513] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.518342 2896 leader_election.cc:290] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.518352 1154 raft_consensus.cc:357] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.518517 1154 raft_consensus.cc:383] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.518601 1154 raft_consensus.cc:738] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.519017 1154 consensus_queue.cc:260] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.522308 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6f7051dc43b84d7bbf70b51f0b9bced8
I20250629 01:52:03.526091 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.526364 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9ea65305417e45fdb1a47768cc1ee054
I20250629 01:52:03.526666 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3d8b63e73ce543d2bbfa915ed1f776fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.526916 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0dac5dbb476a45639f013c63b022f815"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.527145 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d8b63e73ce543d2bbfa915ed1f776fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:03.527303 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dac5dbb476a45639f013c63b022f815 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.527381 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ea65305417e45fdb1a47768cc1ee054 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.528317 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f42144c735164b209ece20fb363c95dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.528636 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f42144c735164b209ece20fb363c95dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.518882 1159 consensus_queue.cc:260] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:03.528715 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e16e6adfa2be40a2885267800331f664 consensus state has no leader
W20250629 01:52:03.533931 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet abbefa9110cc41989c33d25e3ee8ea07 consensus state has no leader
I20250629 01:52:03.541044 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ec3937679b37476f934aceb30ac62d68 (table default.Table_9200a719f8584b2899fbb70d1be7843c [id=786428b4d32f438f8ac080af885b50a4]) successfully deleted
I20250629 01:52:03.544476 453 catalog_manager.cc:5582] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.545349 453 catalog_manager.cc:5582] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:03.545727 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d6b6d5ff92c343cb8f1b3039b84390af (table default.Table_9c12a09abc39412497235b71efb0ea8e [id=8038bbd131674ed59dbe36c73c14d466]). Expected version 3 got 0
I20250629 01:52:03.546025 1159 ts_tablet_manager.cc:1428] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.045s user 0.001s sys 0.000s
I20250629 01:52:03.519589 1154 ts_tablet_manager.cc:1428] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:03.547803 1154 tablet_bootstrap.cc:492] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.550187 912 raft_consensus.cc:3058] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.520007 2758 raft_consensus.cc:491] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.553617 2758 raft_consensus.cc:513] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.554102 2758 leader_election.cc:290] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.538056 2760 consensus_queue.cc:1035] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250629 01:52:03.539381 1158 tablet_replica.cc:331] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.529402 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ea65305417e45fdb1a47768cc1ee054 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.554627 1158 raft_consensus.cc:2241] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.554751 1158 raft_consensus.cc:2270] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.555187 1158 ts_tablet_manager.cc:1905] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.558064 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe1f8083a884cdd97afce83bd22073c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.558219 772 raft_consensus.cc:2466] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.561040 1155 tablet_bootstrap.cc:492] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.561244 1155 ts_tablet_manager.cc:1397] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.007s
I20250629 01:52:03.524879 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.561370 986 leader_election.cc:304] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:03.561776 2760 raft_consensus.cc:2802] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.561856 2760 raft_consensus.cc:491] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.561920 2760 raft_consensus.cc:3058] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.561915 1155 raft_consensus.cc:357] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.562032 1155 raft_consensus.cc:383] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.562067 1155 raft_consensus.cc:738] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.562227 1155 consensus_queue.cc:260] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.563254 1155 ts_tablet_manager.cc:1428] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:03.563509 1155 tablet_bootstrap.cc:492] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.564496 2758 raft_consensus.cc:491] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.564638 2758 raft_consensus.cc:513] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.565095 1155 tablet_bootstrap.cc:654] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.565187 2758 leader_election.cc:290] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.565716 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cced2e8c94345b88f388becb0060d7a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:03.566126 986 leader_election.cc:343] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.566291 1154 tablet_bootstrap.cc:654] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.569646 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:03.571033 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_9e68c12ac74247ffb80d0ee3b879d226 [id=c6251e55d21e4558a16574655b0be5c7] alter complete (version 2)
W20250629 01:52:03.536958 850 leader_election.cc:343] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 3bdc6a26b3594d6fb33a27cac5b29f4f
I20250629 01:52:03.573654 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.574940 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_d9c2bb1acc984b17a6f7ea9ed229be68 [id=4bbe205f87d54d69821df73deff787e0] alter complete (version 2)
I20250629 01:52:03.576936 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.577351 744 tablet_service.cc:1515] Processing DeleteTablet for tablet ef3060d46ac24330be49103920dc1dee with delete_type TABLET_DATA_DELETED (replaced by tablet 86099eb9e40a4ea389afc059f90a97ef) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.578131 1155 tablet_bootstrap.cc:492] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.578284 1155 ts_tablet_manager.cc:1397] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250629 01:52:03.578474 1159 tablet_bootstrap.cc:492] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.579221 1155 raft_consensus.cc:357] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.579406 1155 raft_consensus.cc:383] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.579500 1155 raft_consensus.cc:738] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.579728 1155 consensus_queue.cc:260] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.580189 1155 ts_tablet_manager.cc:1428] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:03.580451 1155 tablet_bootstrap.cc:492] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.581754 1155 tablet_bootstrap.cc:654] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.581923 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet f465d0bb46674763abfc8d93f45cd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.582355 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet bccdcd9de08648bb925db6fe56f97d20 with delete_type TABLET_DATA_DELETED (replaced by tablet fb48f4a2d9724300b9c00df6e12e92b0) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.582815 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.583218 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet eb26725bd175429d805947c4eda6e40a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.583696 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ea65305417e45fdb1a47768cc1ee054 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.584043 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 3bdc6a26b3594d6fb33a27cac5b29f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:03.584440 850 leader_election.cc:343] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 3bdc6a26b3594d6fb33a27cac5b29f4f
I20250629 01:52:03.584578 850 leader_election.cc:304] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.584790 2896 raft_consensus.cc:2747] T 3bdc6a26b3594d6fb33a27cac5b29f4f P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.585891 1156 ts_tablet_manager.cc:1918] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.585973 1156 log.cc:1199] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/7908afa2abaa4521b44f954e0790f986
I20250629 01:52:03.586074 1159 tablet_bootstrap.cc:654] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.586189 1156 ts_tablet_manager.cc:1939] T 7908afa2abaa4521b44f954e0790f986 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.588883 1158 ts_tablet_manager.cc:1918] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.588989 1158 log.cc:1199] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/bb98aa9d365541969ea1dc70960760b2
I20250629 01:52:03.589249 1158 ts_tablet_manager.cc:1939] T bb98aa9d365541969ea1dc70960760b2 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.590510 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:03.591163 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1416) took 1707 ms. Trace:
I20250629 01:52:03.591234 1156 rpcz_store.cc:276] 0629 01:52:01.883333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.932678 (+ 49345us) service_pool.cc:225] Handling call
0629 01:52:03.490193 (+1557515us) ts_tablet_manager.cc:1145] Deleting tablet 7908afa2abaa4521b44f954e0790f986
0629 01:52:03.490202 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.508281 (+ 18079us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.585799 (+ 77518us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.591138 (+ 5339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1557271}
I20250629 01:52:03.591480 1156 tablet_replica.cc:331] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:03.591616 1156 raft_consensus.cc:2241] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.591709 1156 raft_consensus.cc:2270] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.591909 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cced2e8c94345b88f388becb0060d7a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.592080 1156 ts_tablet_manager.cc:1905] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:03.592723 988 leader_election.cc:343] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.592809 988 leader_election.cc:304] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.592800 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe1f8083a884cdd97afce83bd22073c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.593093 2758 raft_consensus.cc:2747] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.594022 988 leader_election.cc:343] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.595247 2760 raft_consensus.cc:513] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.595685 2760 leader_election.cc:290] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.596264 2855 tablet.cc:1722] T a12fffe61d7140308acd181bfa3d2700 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:03.597572 1154 tablet_bootstrap.cc:492] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.598039 1154 ts_tablet_manager.cc:1397] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.002s
I20250629 01:52:03.598052 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 7908afa2abaa4521b44f954e0790f986 (table default.Table_e998111141ca474dbff482e945a67465 [id=a3739e0c0f154fec8a5ffbf6b2b08fcc]) successfully deleted
I20250629 01:52:03.598521 1155 tablet_bootstrap.cc:492] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.598675 1155 ts_tablet_manager.cc:1397] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250629 01:52:03.598691 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe1f8083a884cdd97afce83bd22073c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.598827 772 raft_consensus.cc:3058] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.598819 1154 raft_consensus.cc:357] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.598893 2859 tablet_replica.cc:331] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:03.598935 1154 raft_consensus.cc:383] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.598980 1154 raft_consensus.cc:738] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.599116 2859 raft_consensus.cc:2241] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.599131 1154 consensus_queue.cc:260] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.599229 2859 raft_consensus.cc:2270] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.599423 1155 raft_consensus.cc:357] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.599555 1155 raft_consensus.cc:383] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.599596 1155 raft_consensus.cc:738] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.599624 2859 ts_tablet_manager.cc:1905] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.599781 1155 consensus_queue.cc:260] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.600198 1155 ts_tablet_manager.cc:1428] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:03.600699 1154 ts_tablet_manager.cc:1428] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:03.600759 1155 tablet_bootstrap.cc:492] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.600908 1154 tablet_bootstrap.cc:492] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.601970 1154 tablet_bootstrap.cc:654] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.602059 1155 tablet_bootstrap.cc:654] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.606611 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bbe1f8083a884cdd97afce83bd22073c" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.606667 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet daf312e60c014fe698d3ec9f1a1b769c (table default.Table_3f1b07c24c6441f4a4965f8e055648fe [id=df4a1725208848efa85ea19a1d3da531]) successfully deleted
I20250629 01:52:03.608364 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a12fffe61d7140308acd181bfa3d2700 (table default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991]). Expected version 1 got 0
I20250629 01:52:03.609012 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5bbd9c09a3a2490ab18ec643b60078b1" } modify_external_catalogs: true
I20250629 01:52:03.611001 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 0323947b135a4308903b1eed1ca4db71 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:03.609134 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5bbd9c09a3a2490ab18ec643b60078b1" } modify_external_catalogs: true
I20250629 01:52:03.616895 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 0323947b135a4308903b1eed1ca4db71
I20250629 01:52:03.617244 447 catalog_manager.cc:6652] ReplaceTablet: tablet 0323947b135a4308903b1eed1ca4db71 deleted and replaced by tablet d081b4e3bb094674a9ff4faf5182e021
I20250629 01:52:03.618325 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 0323947b135a4308903b1eed1ca4db71 with delete_type TABLET_DATA_DELETED (replaced by tablet d081b4e3bb094674a9ff4faf5182e021) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.619082 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 0323947b135a4308903b1eed1ca4db71 with delete_type TABLET_DATA_DELETED (replaced by tablet d081b4e3bb094674a9ff4faf5182e021) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.620386 1155 tablet_bootstrap.cc:492] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.620498 1155 ts_tablet_manager.cc:1397] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250629 01:52:03.621102 1155 raft_consensus.cc:357] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.621286 1155 raft_consensus.cc:383] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.621373 1155 raft_consensus.cc:738] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.621611 1155 consensus_queue.cc:260] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.621702 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 5db79f2d97ec4c87931522aa4c27965f
I20250629 01:52:03.622206 1155 ts_tablet_manager.cc:1428] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:52:03.622440 1155 tablet_bootstrap.cc:492] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.623790 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5db79f2d97ec4c87931522aa4c27965f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.623891 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_d759086c226f4bb0a9aa12de777505e2 [id=075cbade2bd84a68a76f354b7796c991] alter complete (version 1)
I20250629 01:52:03.624354 1155 tablet_bootstrap.cc:654] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.625994 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 5db79f2d97ec4c87931522aa4c27965f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.626317 879 tablet_service.cc:1515] Processing DeleteTablet for tablet ce90c045115844b0b96143f91f1be1d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.631706 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c525edb71817495cbedf4ec757b7810e" } modify_external_catalogs: true
I20250629 01:52:03.631831 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c525edb71817495cbedf4ec757b7810e" } modify_external_catalogs: true
I20250629 01:52:03.632817 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_892a1c3fe5b6433bab9788f706a4dd74" } new_table_name: "default.Table_ed23aba5f178454b9ea8a990a1852ef0" modify_external_catalogs: true
I20250629 01:52:03.633725 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9d65bac066b8474f8f1166914053b2a4" } new_table_name: "default.Table_31825683c00b4327866114e089fb02d7" modify_external_catalogs: true
I20250629 01:52:03.642952 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet e5e96c7df4b2470a89ef0ef8e9e97950
I20250629 01:52:03.643505 744 tablet_service.cc:1515] Processing DeleteTablet for tablet e5e96c7df4b2470a89ef0ef8e9e97950 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.644587 1155 tablet_bootstrap.cc:492] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.644744 1155 ts_tablet_manager.cc:1397] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250629 01:52:03.645328 1155 raft_consensus.cc:357] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:52:03.645483 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a43f6594e4844af580c99ea598272aae consensus state has no leader
I20250629 01:52:03.645512 1155 raft_consensus.cc:383] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.645556 1155 raft_consensus.cc:738] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.645771 1155 consensus_queue.cc:260] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
W20250629 01:52:03.646076 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9bbff29d38bf437da3c8273ba1bbec3f consensus state has no leader
I20250629 01:52:03.646557 879 tablet_service.cc:1515] Processing DeleteTablet for tablet e5e96c7df4b2470a89ef0ef8e9e97950 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.646608 1155 ts_tablet_manager.cc:1428] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:03.646615 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_36921981749241789ba5b79d109b0549" } modify_external_catalogs: true
I20250629 01:52:03.646721 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_36921981749241789ba5b79d109b0549" } modify_external_catalogs: true
I20250629 01:52:03.647099 1155 tablet_bootstrap.cc:492] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.648568 1155 tablet_bootstrap.cc:654] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.653133 453 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a43f6594e4844af580c99ea598272aae (table default.Table_31825683c00b4327866114e089fb02d7 [id=c1135b07e9bd455cb21ae0061c94e017]). Expected version 3 got 0
I20250629 01:52:03.654341 1159 tablet_bootstrap.cc:492] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.654460 1159 ts_tablet_manager.cc:1397] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.001s
I20250629 01:52:03.655045 1159 raft_consensus.cc:357] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.655185 1159 raft_consensus.cc:383] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.655236 1159 raft_consensus.cc:738] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.655427 1159 consensus_queue.cc:260] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.656556 2896 raft_consensus.cc:491] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.656654 2896 raft_consensus.cc:513] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.657087 2896 leader_election.cc:290] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.604616 2852 tablet.cc:1722] T 3d87639255ec46d98fcfb0f9638ee39a P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250629 01:52:03.614248 988 leader_election.cc:343] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.657835 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 43587.5us
I20250629 01:52:03.618736 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 0323947b135a4308903b1eed1ca4db71 with delete_type TABLET_DATA_DELETED (replaced by tablet d081b4e3bb094674a9ff4faf5182e021) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.623554 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 5db79f2d97ec4c87931522aa4c27965f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.627133 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.644132 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet e5e96c7df4b2470a89ef0ef8e9e97950 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.662233 1158 tablet_replica.cc:331] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.662395 1158 raft_consensus.cc:2241] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.662510 1158 raft_consensus.cc:2270] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.662570 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_43f2d7b5617b45c9a3acc51f0add89df" } new_table_name: "default.Table_92b822d0311145139470e046c88715bf" modify_external_catalogs: true
I20250629 01:52:03.663045 1158 ts_tablet_manager.cc:1905] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.664108 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet bb98aa9d365541969ea1dc70960760b2 (table default.Table_9790ad1c1a4447949cfe0e98c46458df [id=631e5b6c35a7419a8963c0a4e468e439]) successfully deleted
I20250629 01:52:03.665128 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_96dae305586a4e209ec51d48ff898fe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.665540 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96dae305586a4e209ec51d48ff898fe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.666456 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_35d900b579bb493d9bd7cc39c09da9be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.666642 850 leader_election.cc:343] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ce90c045115844b0b96143f91f1be1d4
W20250629 01:52:03.666834 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d900b579bb493d9bd7cc39c09da9be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.667148 1159 ts_tablet_manager.cc:1428] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250629 01:52:03.668709 1159 tablet_bootstrap.cc:492] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:03.671147 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb48f4a2d9724300b9c00df6e12e92b0 consensus state has no leader
W20250629 01:52:03.672683 850 leader_election.cc:343] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: ce90c045115844b0b96143f91f1be1d4
I20250629 01:52:03.672772 850 leader_election.cc:304] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.673221 2896 raft_consensus.cc:2747] T ce90c045115844b0b96143f91f1be1d4 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.673591 988 consensus_peers.cc:489] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.675624 2895 raft_consensus.cc:491] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.675724 2895 raft_consensus.cc:513] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.676229 2895 leader_election.cc:290] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.672065 776 raft_consensus.cc:1273] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Refusing update from remote peer 4712643b602e4f898752fcab6a1e5913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250629 01:52:03.676614 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43f6594e4844af580c99ea598272aae" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:03.679073 986 leader_election.cc:343] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.679347 2895 consensus_queue.cc:1035] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:03.680289 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43f6594e4844af580c99ea598272aae" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.681274 988 leader_election.cc:343] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.681377 988 leader_election.cc:304] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.681829 2895 raft_consensus.cc:2747] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.682991 1159 tablet_bootstrap.cc:654] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.685421 772 raft_consensus.cc:2466] T bbe1f8083a884cdd97afce83bd22073c P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:03.688330 986 leader_election.cc:304] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:03.688513 986 raft_consensus.cc:2684] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:52:03.691572 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:03.703902 744 tablet_service.cc:1515] Processing DeleteTablet for tablet ec3937679b37476f934aceb30ac62d68 with delete_type TABLET_DATA_DELETED (replaced by tablet 966e379adef8435887a2d8f39bb99968) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.704463 744 tablet_service.cc:1515] Processing DeleteTablet for tablet bbe1f8083a884cdd97afce83bd22073c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:59 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.705394 2896 raft_consensus.cc:491] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.705509 2896 raft_consensus.cc:513] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.706061 2896 leader_election.cc:290] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:03.706768 850 leader_election.cc:343] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 1b0782c7e652419096710563930f2072
W20250629 01:52:03.707320 850 leader_election.cc:343] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 1b0782c7e652419096710563930f2072
I20250629 01:52:03.707413 850 leader_election.cc:304] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.707604 2896 raft_consensus.cc:2747] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.708353 1159 tablet_bootstrap.cc:492] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.708465 1159 ts_tablet_manager.cc:1397] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.001s
I20250629 01:52:03.709121 1159 raft_consensus.cc:357] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.709250 1159 raft_consensus.cc:383] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.709295 1159 raft_consensus.cc:738] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.709606 1159 consensus_queue.cc:260] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.713413 1159 ts_tablet_manager.cc:1428] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:52:03.714283 1159 tablet_bootstrap.cc:492] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.716037 1159 tablet_bootstrap.cc:654] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.719089 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 8295e8178eff4d5eb95ac885092731e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:45 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.720605 1155 tablet_bootstrap.cc:492] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.720767 1155 ts_tablet_manager.cc:1397] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.074s user 0.005s sys 0.009s
I20250629 01:52:03.721379 1155 raft_consensus.cc:357] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.721585 1155 raft_consensus.cc:383] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.721661 1155 raft_consensus.cc:738] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.721892 1155 consensus_queue.cc:260] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.722338 1155 ts_tablet_manager.cc:1428] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:03.722609 1155 tablet_bootstrap.cc:492] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.724000 1155 tablet_bootstrap.cc:654] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.725133 1154 tablet_bootstrap.cc:492] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.725302 1154 ts_tablet_manager.cc:1397] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.008s
I20250629 01:52:03.725951 1154 raft_consensus.cc:357] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.726027 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7ca2e46142c5424ab8f471c13d635ca2" } modify_external_catalogs: true
I20250629 01:52:03.726155 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7ca2e46142c5424ab8f471c13d635ca2" } modify_external_catalogs: true
I20250629 01:52:03.726356 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8e651a7d91094b0d9f5526c179c6205d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.727030 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e651a7d91094b0d9f5526c179c6205d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.728245 1154 raft_consensus.cc:383] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.729331 2896 raft_consensus.cc:491] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.729432 2896 raft_consensus.cc:513] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.729918 2896 leader_election.cc:290] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
I20250629 01:52:03.730012 1154 raft_consensus.cc:738] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:52:03.730360 712 consensus_peers.cc:489] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.730346 1154 consensus_queue.cc:260] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.731536 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_983d9cfd606940e1815eadca20ef3ecc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.731639 850 leader_election.cc:343] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 8295e8178eff4d5eb95ac885092731e0
W20250629 01:52:03.731962 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_983d9cfd606940e1815eadca20ef3ecc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.732290 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_20da2869379148dba4a28145a932483b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.732684 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20da2869379148dba4a28145a932483b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.733534 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_292a95f97cbd424ea5aba15748ff9ee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.733592 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2831a79356254e00b88ca2149d61af2c" } new_table_name: "default.Table_aafacf904f5d40de8ad9f8f2c0718b76" modify_external_catalogs: true
W20250629 01:52:03.733875 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_292a95f97cbd424ea5aba15748ff9ee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.734119 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_11a668da1f204d02b70b4ccd24b59bac" } modify_external_catalogs: true
I20250629 01:52:03.734264 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_11a668da1f204d02b70b4ccd24b59bac" } modify_external_catalogs: true
I20250629 01:52:03.737120 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1b3364a7417244a59e0dc1cae5101612" } new_table_name: "default.Table_28ac322ff4554d4f96a747dd53ead444" modify_external_catalogs: true
I20250629 01:52:03.740937 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 2045012c80d049eaabf7dec1df28b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.748821 1155 tablet_bootstrap.cc:492] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.748983 1155 ts_tablet_manager.cc:1397] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
I20250629 01:52:03.749375 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0710fe46a5274c88bdbbc51b5cc28aa9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.749747 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0710fe46a5274c88bdbbc51b5cc28aa9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.749698 1155 raft_consensus.cc:357] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.749831 1155 raft_consensus.cc:383] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.749879 1155 raft_consensus.cc:738] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.750067 1155 consensus_queue.cc:260] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.750463 1155 ts_tablet_manager.cc:1428] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:52:03.750535 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_1241f1bbd284404885404a1fe5c5a570"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:03.750705 1155 tablet_bootstrap.cc:492] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:03.750898 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1241f1bbd284404885404a1fe5c5a570 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.751540 2896 raft_consensus.cc:491] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.751657 2896 raft_consensus.cc:513] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.752146 2896 leader_election.cc:290] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:03.753958 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a873bc03533a405185eca6ad4bb5db3a consensus state has no leader
I20250629 01:52:03.754154 1159 tablet_bootstrap.cc:492] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.754253 1159 ts_tablet_manager.cc:1397] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20250629 01:52:03.754221 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0f315136e0174620a55372819dd1c75b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.754551 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f315136e0174620a55372819dd1c75b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.755066 1159 raft_consensus.cc:357] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.755187 1159 raft_consensus.cc:383] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.755228 1159 raft_consensus.cc:738] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.755676 1159 consensus_queue.cc:260] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:03.756765 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 189c6c3627404c2685d77cb3b12cace8 consensus state has no leader
I20250629 01:52:03.758415 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0af022a022f8409c8373bc998d7ed733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.758754 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0af022a022f8409c8373bc998d7ed733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.759631 1159 ts_tablet_manager.cc:1428] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250629 01:52:03.760231 1155 tablet_bootstrap.cc:654] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.760610 1159 tablet_bootstrap.cc:492] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:03.761817 850 leader_election.cc:343] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 8295e8178eff4d5eb95ac885092731e0
I20250629 01:52:03.761914 850 leader_election.cc:304] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.762108 2896 raft_consensus.cc:2747] T 8295e8178eff4d5eb95ac885092731e0 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.762267 850 leader_election.cc:343] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e4d6f1eb43b24fda94aa0afcf4501195
W20250629 01:52:03.762428 850 leader_election.cc:343] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e4d6f1eb43b24fda94aa0afcf4501195
I20250629 01:52:03.762498 850 leader_election.cc:304] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.762670 2896 raft_consensus.cc:2747] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.764890 1154 ts_tablet_manager.cc:1428] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250629 01:52:03.765162 1154 tablet_bootstrap.cc:492] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.766474 1154 tablet_bootstrap.cc:654] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.771430 2852 tablet.cc:1722] T d6b6d5ff92c343cb8f1b3039b84390af P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:52:03.772992 1155 tablet_bootstrap.cc:492] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.773103 1155 ts_tablet_manager.cc:1397] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20250629 01:52:03.773741 1155 raft_consensus.cc:357] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.773861 1155 raft_consensus.cc:383] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.773905 1155 raft_consensus.cc:738] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.774080 1155 consensus_queue.cc:260] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.774456 1155 ts_tablet_manager.cc:1428] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250629 01:52:03.774679 1155 tablet_bootstrap.cc:492] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.775692 2908 tablet.cc:1722] T d6b6d5ff92c343cb8f1b3039b84390af P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250629 01:52:03.776041 1155 tablet_bootstrap.cc:654] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.781389 1156 ts_tablet_manager.cc:1918] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.781474 1156 log.cc:1199] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/07e0727102d54294a7c13a4aeef311e1
I20250629 01:52:03.781697 1156 ts_tablet_manager.cc:1939] T 07e0727102d54294a7c13a4aeef311e1 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:03.793057 1159 tablet_bootstrap.cc:654] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.793376 1154 tablet_bootstrap.cc:492] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:03.793483 1154 ts_tablet_manager.cc:1397] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.008s
I20250629 01:52:03.794189 1154 raft_consensus.cc:357] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.794338 1154 raft_consensus.cc:383] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.794394 1154 raft_consensus.cc:738] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.794557 1154 consensus_queue.cc:260] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.794998 2894 consensus_queue.cc:1035] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:03.802549 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 303f77a729c648f9b1146d11056b242d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.807567 2855 tablet.cc:1722] T b15300fd8ca7489f9d77f1f8992f8df3 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:03.824149 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d2750afeab44fda88c390b72b529bcc" } modify_external_catalogs: true
I20250629 01:52:03.795007 1154 ts_tablet_manager.cc:1428] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250629 01:52:03.824455 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_2d2750afeab44fda88c390b72b529bcc" } modify_external_catalogs: true
I20250629 01:52:03.830634 1154 tablet_bootstrap.cc:492] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:03.840065 2907 raft_consensus.cc:491] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.840245 2907 raft_consensus.cc:513] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.840895 2907 leader_election.cc:290] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.841249 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef3060d46ac24330be49103920dc1dee" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.841789 712 leader_election.cc:343] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.841790 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef3060d46ac24330be49103920dc1dee" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:03.841945 1052 raft_consensus.cc:2466] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:03.842327 710 leader_election.cc:304] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.842675 2907 raft_consensus.cc:2802] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.842769 2907 raft_consensus.cc:491] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.842859 2907 raft_consensus.cc:3058] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.845490 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1241f1bbd284404885404a1fe5c5a570" } modify_external_catalogs: true
I20250629 01:52:03.845636 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1241f1bbd284404885404a1fe5c5a570" } modify_external_catalogs: true
I20250629 01:52:03.846755 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d6fad403f1ae4427895f31b28a95daf3" } new_table_name: "default.Table_5beb37ebcae94bf296cb0d46056f7720" modify_external_catalogs: true
I20250629 01:52:03.848461 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aa8f14fe50764434b6dfb679cf7a0c71" } new_table_name: "default.Table_fc5339fa96804212b52a49b94f23728b" modify_external_catalogs: true
I20250629 01:52:03.849041 2935 tablet.cc:1722] T b15300fd8ca7489f9d77f1f8992f8df3 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250629 01:52:03.850997 988 consensus_peers.cc:489] T a582d4f907f340179024c0377d3893e2 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.851023 1155 tablet_bootstrap.cc:492] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.851136 1155 ts_tablet_manager.cc:1397] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.010s
I20250629 01:52:03.851843 1155 raft_consensus.cc:357] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.851989 1155 raft_consensus.cc:383] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.852035 1155 raft_consensus.cc:738] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.852205 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bfff062c5d7542b2b72f24c47c1fb311
I20250629 01:52:03.852250 1155 consensus_queue.cc:260] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.852816 1155 ts_tablet_manager.cc:1428] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:52:03.853058 1155 tablet_bootstrap.cc:492] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:03.853531 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 849e3e59b9624994bbb63027aa85d0b0 consensus state has no leader
I20250629 01:52:03.854920 1155 tablet_bootstrap.cc:654] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:03.857154 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e16e6adfa2be40a2885267800331f664 consensus state has no leader
I20250629 01:52:03.860641 2758 raft_consensus.cc:491] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.860805 2758 raft_consensus.cc:513] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.861359 2907 raft_consensus.cc:513] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.862016 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef3060d46ac24330be49103920dc1dee" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:03.862008 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c7b76cb56b24604b13ee29c7f576ce2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.862144 1052 raft_consensus.cc:3058] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.862457 2534 raft_consensus.cc:491] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.862566 2534 raft_consensus.cc:513] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:03.862594 988 leader_election.cc:343] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.862780 2758 leader_election.cc:290] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.862979 2534 leader_election.cc:290] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.863328 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "021da4395e414413b9b11ebcfbffcd19" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.863744 712 leader_election.cc:343] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.864203 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "021da4395e414413b9b11ebcfbffcd19" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:03.864552 2907 leader_election.cc:290] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:03.864602 710 leader_election.cc:343] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.864691 710 leader_election.cc:304] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:03.864925 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef3060d46ac24330be49103920dc1dee" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.864971 2907 raft_consensus.cc:2747] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.865191 1052 raft_consensus.cc:2466] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:03.865326 712 leader_election.cc:343] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.865514 710 leader_election.cc:304] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.865660 2859 ts_tablet_manager.cc:1918] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.865824 2859 log.cc:1199] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/3a81e6f9711c4d398ae09f33a75566ae
I20250629 01:52:03.865870 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c7b76cb56b24604b13ee29c7f576ce2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.866012 776 raft_consensus.cc:2466] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.866083 2859 ts_tablet_manager.cc:1939] T 3a81e6f9711c4d398ae09f33a75566ae P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:03.866324 986 leader_election.cc:304] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.866827 2760 raft_consensus.cc:2802] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.866829 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1417) took 1983 ms. Trace:
I20250629 01:52:03.866950 2760 raft_consensus.cc:491] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.866954 2534 raft_consensus.cc:2802] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:03.867065 2534 raft_consensus.cc:695] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:03.867094 2760 raft_consensus.cc:3058] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.867283 2534 consensus_queue.cc:237] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.866901 1156 rpcz_store.cc:276] 0629 01:52:01.883424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:01.939910 (+ 56486us) service_pool.cc:225] Handling call
0629 01:52:03.591441 (+1651531us) ts_tablet_manager.cc:1145] Deleting tablet 07e0727102d54294a7c13a4aeef311e1
0629 01:52:03.591450 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.682708 (+ 91258us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.781298 (+ 98590us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.866813 (+ 85515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1650139}
I20250629 01:52:03.868710 1156 tablet_replica.cc:331] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:03.868887 1156 raft_consensus.cc:2241] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:03.869139 1156 raft_consensus.cc:2270] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.869565 1156 ts_tablet_manager.cc:1905] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.870055 1158 ts_tablet_manager.cc:1918] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:03.870149 1158 log.cc:1199] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/bbe1f8083a884cdd97afce83bd22073c
I20250629 01:52:03.870374 1158 ts_tablet_manager.cc:1939] T bbe1f8083a884cdd97afce83bd22073c P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:03.872826 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_cbb81a42b45040139dcda6493db2fe5f" } new_table_name: "default.Table_d45cec6c727f4b9d99242fd86a97e9ea" modify_external_catalogs: true
I20250629 01:52:03.873641 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 07e0727102d54294a7c13a4aeef311e1 (table default.Table_a9fddce63afb42b4be73a8c4807e050a [id=d8e51d2596fa4215bd2e2a3c43c466f1]) successfully deleted
I20250629 01:52:03.875473 2548 raft_consensus.cc:491] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.875574 2548 raft_consensus.cc:513] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.876133 2548 leader_election.cc:290] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.876478 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec3937679b37476f934aceb30ac62d68" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.876917 712 leader_election.cc:343] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.877005 710 leader_election.cc:343] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ec3937679b37476f934aceb30ac62d68
I20250629 01:52:03.877060 712 leader_election.cc:304] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:03.878124 453 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet d6b6d5ff92c343cb8f1b3039b84390af (table default.Table_9c12a09abc39412497235b71efb0ea8e [id=8038bbd131674ed59dbe36c73c14d466]). Expected version 3 got 0
I20250629 01:52:03.878971 2548 raft_consensus.cc:2747] T ec3937679b37476f934aceb30ac62d68 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.879560 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d3f10c12a5284e1aa89799a41ea487dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.879978 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3f10c12a5284e1aa89799a41ea487dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.880807 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_849da15438e74c09b66190ea520f89bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.881172 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_849da15438e74c09b66190ea520f89bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.881568 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 01d95356c7a7419bb56ddf67d451eadc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.882210 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 021da4395e414413b9b11ebcfbffcd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.882567 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c7b76cb56b24604b13ee29c7f576ce2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.883275 744 tablet_service.cc:1515] Processing DeleteTablet for tablet bccdcd9de08648bb925db6fe56f97d20 with delete_type TABLET_DATA_DELETED (replaced by tablet fb48f4a2d9724300b9c00df6e12e92b0) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.883635 2534 consensus_queue.cc:1035] T d7e2c70221e94923b6a324a03f7133c6 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:03.886389 1155 tablet_bootstrap.cc:492] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.886530 1155 ts_tablet_manager.cc:1397] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.003s
I20250629 01:52:03.887176 1155 raft_consensus.cc:357] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.887326 1155 raft_consensus.cc:383] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.887400 1155 raft_consensus.cc:738] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.887585 1155 consensus_queue.cc:260] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.888448 447 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:03.889107 1155 ts_tablet_manager.cc:1428] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:03.890255 1155 tablet_bootstrap.cc:492] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
W20250629 01:52:03.891180 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3db646a559340269980306633f26965 consensus state has no leader
I20250629 01:52:03.891737 1155 tablet_bootstrap.cc:654] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.891801 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 29e05dd9908c4390a55b55e2eef8d4b2
I20250629 01:52:03.892457 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 29e05dd9908c4390a55b55e2eef8d4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.893309 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 29e05dd9908c4390a55b55e2eef8d4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.893411 1159 tablet_bootstrap.cc:492] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.893553 1159 ts_tablet_manager.cc:1397] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.133s user 0.002s sys 0.000s
I20250629 01:52:03.893718 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 29e05dd9908c4390a55b55e2eef8d4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.894266 1159 raft_consensus.cc:357] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.894414 1159 raft_consensus.cc:383] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.894474 1159 raft_consensus.cc:738] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.894672 1159 consensus_queue.cc:260] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.895134 1159 ts_tablet_manager.cc:1428] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:52:03.898133 1159 tablet_bootstrap.cc:492] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.898662 2896 raft_consensus.cc:491] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.898753 2896 raft_consensus.cc:513] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.899209 2896 leader_election.cc:290] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891)
W20250629 01:52:03.899926 850 leader_election.cc:343] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 303f77a729c648f9b1146d11056b242d
W20250629 01:52:03.900218 850 leader_election.cc:343] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 303f77a729c648f9b1146d11056b242d
I20250629 01:52:03.900291 850 leader_election.cc:304] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.900897 2896 raft_consensus.cc:2747] T 303f77a729c648f9b1146d11056b242d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:03.903762 2907 raft_consensus.cc:491] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.903888 2907 raft_consensus.cc:513] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.904384 2907 leader_election.cc:290] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:03.904628 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bccdcd9de08648bb925db6fe56f97d20" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.905026 712 leader_election.cc:343] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.905498 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 7908afa2abaa4521b44f954e0790f986 with delete_type TABLET_DATA_DELETED (replaced by tablet 4af33b24ea404b1ab2608ea2e12b52a8) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.905537 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bccdcd9de08648bb925db6fe56f97d20" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:03.905671 1048 raft_consensus.cc:2466] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:03.907151 710 leader_election.cc:304] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.907516 2907 raft_consensus.cc:2802] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.907581 2907 raft_consensus.cc:491] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.907634 2907 raft_consensus.cc:3058] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.912976 1155 tablet_bootstrap.cc:492] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.913102 1155 ts_tablet_manager.cc:1397] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20250629 01:52:03.914064 1155 raft_consensus.cc:357] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.914242 1155 raft_consensus.cc:383] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.914286 1155 raft_consensus.cc:738] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.914487 1155 consensus_queue.cc:260] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.915828 1155 ts_tablet_manager.cc:1428] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:52:03.916785 1155 tablet_bootstrap.cc:492] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.918301 1155 tablet_bootstrap.cc:654] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.920608 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 35a62c3fe4fb4220838deece8e46d074
I20250629 01:52:03.921564 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 4ba02adec0d74547ab12dff3a0c898f7
I20250629 01:52:03.922632 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 2b3045aa142e4471a79de74a5a272e36
I20250629 01:52:03.923276 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_9c12a09abc39412497235b71efb0ea8e [id=8038bbd131674ed59dbe36c73c14d466] alter complete (version 3)
I20250629 01:52:03.924501 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b3045aa142e4471a79de74a5a272e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.924566 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ba02adec0d74547ab12dff3a0c898f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.924499 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a62c3fe4fb4220838deece8e46d074 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:03.924911 1159 tablet_bootstrap.cc:654] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.932003 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a62c3fe4fb4220838deece8e46d074 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.932476 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 35a62c3fe4fb4220838deece8e46d074 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.932919 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ba02adec0d74547ab12dff3a0c898f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.933349 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 4ba02adec0d74547ab12dff3a0c898f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.934048 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b3045aa142e4471a79de74a5a272e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:03.934478 1159 tablet_bootstrap.cc:492] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.934526 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 2b3045aa142e4471a79de74a5a272e36 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:03 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.934602 1159 ts_tablet_manager.cc:1397] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20250629 01:52:03.935029 744 tablet_service.cc:1515] Processing DeleteTablet for tablet f465d0bb46674763abfc8d93f45cd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.935442 744 tablet_service.cc:1515] Processing DeleteTablet for tablet bb98aa9d365541969ea1dc70960760b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:01 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.935376 1159 raft_consensus.cc:357] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.935511 1159 raft_consensus.cc:383] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.935576 1159 raft_consensus.cc:738] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.935743 1159 consensus_queue.cc:260] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.936007 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ca02bcd9467496c9057b3de9c1b1725 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.936203 1159 ts_tablet_manager.cc:1428] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250629 01:52:03.936491 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 67e90a48af8f48bba3b3fd4f4110d367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:03.937161 1159 tablet_bootstrap.cc:492] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:03.938598 1159 tablet_bootstrap.cc:654] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.944038 2760 raft_consensus.cc:513] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.944581 2760 leader_election.cc:290] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.945259 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c7b76cb56b24604b13ee29c7f576ce2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:03.945662 988 leader_election.cc:343] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.946067 1155 tablet_bootstrap.cc:492] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:03.946241 1155 ts_tablet_manager.cc:1397] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.003s
I20250629 01:52:03.946660 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c7b76cb56b24604b13ee29c7f576ce2" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:03.946805 776 raft_consensus.cc:3058] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.947090 1155 raft_consensus.cc:357] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.947214 1155 raft_consensus.cc:383] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.947265 1155 raft_consensus.cc:738] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.947474 1155 consensus_queue.cc:260] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.948609 1155 ts_tablet_manager.cc:1428] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:03.948872 1159 tablet_bootstrap.cc:492] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:03.949018 1159 ts_tablet_manager.cc:1397] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250629 01:52:03.949327 1155 tablet_bootstrap.cc:492] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:03.949908 1159 raft_consensus.cc:357] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.950639 1155 tablet_bootstrap.cc:654] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.952261 2907 raft_consensus.cc:513] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.952685 2907 leader_election.cc:290] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:03.952690 1159 raft_consensus.cc:383] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:03.952749 1159 raft_consensus.cc:738] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:03.952911 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bccdcd9de08648bb925db6fe56f97d20" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:03.952898 1159 consensus_queue.cc:260] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.953277 1159 ts_tablet_manager.cc:1428] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250629 01:52:03.953486 1159 tablet_bootstrap.cc:492] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:03.953925 712 leader_election.cc:343] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.954232 2760 raft_consensus.cc:491] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.954334 2760 raft_consensus.cc:513] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.954731 1159 tablet_bootstrap.cc:654] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:03.954829 2760 leader_election.cc:290] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.955308 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ed1ee73bb8f4761806a66f988b57bf9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.955654 988 leader_election.cc:343] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.956908 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ed1ee73bb8f4761806a66f988b57bf9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.957034 774 raft_consensus.cc:2466] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.957490 986 leader_election.cc:304] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.958094 2758 raft_consensus.cc:491] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.958191 2758 raft_consensus.cc:513] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.958590 2758 leader_election.cc:290] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.958685 2758 raft_consensus.cc:2802] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.958725 2758 raft_consensus.cc:491] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.958770 2758 raft_consensus.cc:3058] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.961476 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33aa0a66cdbf49c5820e2b43b021f28b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.961686 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9ca0322f970846c9ba95443e38ed67e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.961858 988 leader_election.cc:343] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:03.962157 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ca0322f970846c9ba95443e38ed67e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.962558 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33aa0a66cdbf49c5820e2b43b021f28b" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.963996 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_da26ce67e0ca4692bb783caaf5e530d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.964075 986 leader_election.cc:343] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.964201 986 leader_election.cc:304] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.964504 2760 raft_consensus.cc:2747] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:03.964520 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da26ce67e0ca4692bb783caaf5e530d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.965397 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_dd26153e098741ffbe626a7171016faa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.965894 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd26153e098741ffbe626a7171016faa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.966923 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_3f88c1cdd331456683f3fe4d95aa18af" } new_table_name: "default.Table_42b1bfd299f241f69f9ab0fc3ba89196" modify_external_catalogs: true
I20250629 01:52:03.961488 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1377) took 1019 ms. Trace:
I20250629 01:52:03.968091 2859 rpcz_store.cc:276] 0629 01:52:02.941787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.941840 (+ 53us) service_pool.cc:225] Handling call
0629 01:52:03.598854 (+657014us) ts_tablet_manager.cc:1145] Deleting tablet 3a81e6f9711c4d398ae09f33a75566ae
0629 01:52:03.598860 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.685766 (+ 86906us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.865562 (+179796us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:03.961473 (+ 95911us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":656889}
I20250629 01:52:03.968436 2859 tablet_replica.cc:331] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:03.968595 2859 raft_consensus.cc:2241] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.968705 2859 raft_consensus.cc:2270] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:03.969089 2859 ts_tablet_manager.cc:1905] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:03.969619 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 3a81e6f9711c4d398ae09f33a75566ae (table default.Table_67f686028f0c40c991d3231bd0b0c823 [id=c51c68dd349543d4a4a8bfb1b9aa1184]) successfully deleted
I20250629 01:52:03.970583 454 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:03.971181 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c62354526d4b47b6bbd63c4d14128c5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:03.971513 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c62354526d4b47b6bbd63c4d14128c5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:03.973119 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bccdcd9de08648bb925db6fe56f97d20" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:03.973304 1052 raft_consensus.cc:3058] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.976251 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e646217c593b412f8d30f34145ea5325" } new_table_name: "default.Table_09b1496250654f8583e789cec2305453" modify_external_catalogs: true
I20250629 01:52:03.979305 2760 raft_consensus.cc:491] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.979413 2760 raft_consensus.cc:513] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:03.979842 2760 leader_election.cc:290] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.980286 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5771fdd618e45118df2b6dab7f668d9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:03.980415 774 raft_consensus.cc:2466] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:03.980877 1158 tablet_replica.cc:331] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:03.980880 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5771fdd618e45118df2b6dab7f668d9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:03.981034 1158 raft_consensus.cc:2241] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:03.981140 1158 raft_consensus.cc:2270] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250629 01:52:03.981237 988 leader_election.cc:343] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.981537 986 leader_election.cc:304] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:03.981873 2760 raft_consensus.cc:2802] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:03.981940 2760 raft_consensus.cc:491] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:03.981997 2760 raft_consensus.cc:3058] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:03.981555 1158 ts_tablet_manager.cc:1905] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:03.985700 988 consensus_peers.cc:489] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:03.993422 2894 raft_consensus.cc:491] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:03.993516 2894 raft_consensus.cc:513] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:03.993932 2894 leader_election.cc:290] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:03.994369 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb26725bd175429d805947c4eda6e40a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:03.994792 988 leader_election.cc:343] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.995227 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb26725bd175429d805947c4eda6e40a" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:03.995635 986 leader_election.cc:343] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:03.995713 986 leader_election.cc:304] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:03.996021 2894 raft_consensus.cc:2747] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.000788 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9ed1ee73bb8f4761806a66f988b57bf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.001221 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet bbe1f8083a884cdd97afce83bd22073c (table default.Table_aca8c2c6ca5045838333bfd1e7d46d50 [id=6325ae6cb8a245df89d42a498b2a3828]) successfully deleted
I20250629 01:52:04.014495 448 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:04.017529 879 tablet_service.cc:1468] Processing CreateTablet for tablet e36e7647ca3a4f99bc602a5e2dafffd5 (DEFAULT_TABLE table=default.Table_c65f50be17b4466a9991d2195fef4c5f [id=f7f3250970ec4d218d5d7340d8acb189]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:04.019572 712 consensus_peers.cc:489] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:04.024436 2908 tablet.cc:1722] T b15300fd8ca7489f9d77f1f8992f8df3 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250629 01:52:04.029748 2898 raft_consensus.cc:491] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.029847 2898 raft_consensus.cc:513] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.030252 2898 leader_election.cc:290] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.031427 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ca02bcd9467496c9057b3de9c1b1725" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:04.031571 774 raft_consensus.cc:2466] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:04.032709 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ca02bcd9467496c9057b3de9c1b1725" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:04.033191 986 leader_election.cc:304] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:04.033419 2894 raft_consensus.cc:2802] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.033475 2894 raft_consensus.cc:491] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.033520 2894 raft_consensus.cc:3058] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:04.033898 988 leader_election.cc:343] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.034181 2896 raft_consensus.cc:491] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.034277 2896 raft_consensus.cc:513] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.034541 776 raft_consensus.cc:2466] T 4c7b76cb56b24604b13ee29c7f576ce2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:04.034691 2896 leader_election.cc:290] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:04.034817 2758 raft_consensus.cc:513] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.035241 2758 leader_election.cc:290] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.035204 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.038334 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36e7647ca3a4f99bc602a5e2dafffd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.041106 1156 ts_tablet_manager.cc:1918] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:04.041211 1156 log.cc:1199] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/b6fa908ded8240bb962674e1efb29c74
I20250629 01:52:04.042584 1156 ts_tablet_manager.cc:1939] T b6fa908ded8240bb962674e1efb29c74 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:04.044284 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d062fb830020442faee626be006926c1" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:04.046566 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ed1ee73bb8f4761806a66f988b57bf9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:04.046897 986 leader_election.cc:304] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:04.047091 986 raft_consensus.cc:2684] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250629 01:52:04.047363 2898 consensus_queue.cc:1035] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:04.047821 850 leader_election.cc:304] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.048146 2896 raft_consensus.cc:2747] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.048378 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ed1ee73bb8f4761806a66f988b57bf9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:04.048570 776 raft_consensus.cc:3058] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:04.051076 988 leader_election.cc:343] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:04.054889 988 consensus_peers.cc:489] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:04.058707 2758 consensus_queue.cc:1035] T 4030891de41444e0b387492e570a11a5 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:04.062529 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_39508ad73847431d95213096c80444d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.063046 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39508ad73847431d95213096c80444d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.064343 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9a4a654bd6864178bfacf5ebd390d8d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.064713 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a4a654bd6864178bfacf5ebd390d8d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:04.071120 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac0164ca224e40629bfe5ec6006fba1b consensus state has no leader
I20250629 01:52:04.071403 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_76d78a1d798b4ec5877b83b054814938"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.071732 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76d78a1d798b4ec5877b83b054814938 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.072255 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99304b2c156d4784ba9cb3aed7494313" } modify_external_catalogs: true
I20250629 01:52:04.072384 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_99304b2c156d4784ba9cb3aed7494313" } modify_external_catalogs: true
I20250629 01:52:04.071714 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e88d20b80a994893996a9549bc4dbbb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.073334 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e88d20b80a994893996a9549bc4dbbb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:04.073720 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet feeaa50e7f224da69128ea916e4145fd consensus state has no leader
I20250629 01:52:04.073983 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a8a689877541432b88889c5998bc53fe" } new_table_name: "default.Table_634c80fc6c7d4116b676684a4010317f" modify_external_catalogs: true
I20250629 01:52:04.071714 449 master_service.cc:948] ReplaceTablet: received request to replace tablet cd9a41fae477470281dfba4d4e552b99 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:04.074859 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_d108e95f413c4b36a6fc67336e812ffe [id=7e09582349d64d4fa089b53251ecf9da] alter complete (version 1)
I20250629 01:52:04.075728 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b55682d2903c476d9754ce8cc842594c" } new_table_name: "default.Table_5683602f0dce4ff39355c89a0cb64494" modify_external_catalogs: true
I20250629 01:52:04.076653 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.092870 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9814bb0688d34f939aca5b27c0c27584
I20250629 01:52:04.093171 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet cd9a41fae477470281dfba4d4e552b99
I20250629 01:52:04.093334 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c23a34743c38460887967df74d4a4bfb" } modify_external_catalogs: true
I20250629 01:52:04.093457 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c23a34743c38460887967df74d4a4bfb" } modify_external_catalogs: true
I20250629 01:52:04.093495 449 catalog_manager.cc:6652] ReplaceTablet: tablet cd9a41fae477470281dfba4d4e552b99 deleted and replaced by tablet bf9fddfa57c9414ba95ec28ebd349551
I20250629 01:52:04.094386 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet cd9a41fae477470281dfba4d4e552b99 with delete_type TABLET_DATA_DELETED (replaced by tablet bf9fddfa57c9414ba95ec28ebd349551) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.094518 740 tablet_service.cc:1515] Processing DeleteTablet for tablet cd9a41fae477470281dfba4d4e552b99 with delete_type TABLET_DATA_DELETED (replaced by tablet bf9fddfa57c9414ba95ec28ebd349551) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.094594 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9814bb0688d34f939aca5b27c0c27584 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.094766 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9814bb0688d34f939aca5b27c0c27584 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.095091 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 9814bb0688d34f939aca5b27c0c27584 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.095237 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f14a24eca5f842b19f4361d7bf57557a" } new_table_name: "default.Table_8c5216b202244cbc976dcdac103bee27" modify_external_catalogs: true
W20250629 01:52:04.095739 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de5cf94991af4680ad3efce01b21d94f consensus state has no leader
I20250629 01:52:04.096122 892 tablet_service.cc:1515] Processing DeleteTablet for tablet cd9a41fae477470281dfba4d4e552b99 with delete_type TABLET_DATA_DELETED (replaced by tablet bf9fddfa57c9414ba95ec28ebd349551) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.106053 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 8bb4d7c2096a45df8eb392a2e7d85c3d
W20250629 01:52:04.106384 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3db2d761823340a79987e395e446dcfd has no consensus state
I20250629 01:52:04.107076 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.107290 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.107669 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 8bb4d7c2096a45df8eb392a2e7d85c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:04.119836 440 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ea12171996e6408da1af35e325233e7d (table default.Table_75fe4e5ef4544f3b99c5567decbce1c6 [id=6b38eba78f534170a0401b08b310f122], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ea12171996e6408da1af35e325233e7d consensus state has no leader
I20250629 01:52:04.121985 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f60d91fd7c1949c4ba988f3544ddb755" } modify_external_catalogs: true
I20250629 01:52:04.122120 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f60d91fd7c1949c4ba988f3544ddb755" } modify_external_catalogs: true
I20250629 01:52:04.123878 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c24aa328eefc49d6979d74d3945cc219" } modify_external_catalogs: true
I20250629 01:52:04.124020 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_c24aa328eefc49d6979d74d3945cc219" } modify_external_catalogs: true
I20250629 01:52:04.125664 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0a60f2cb55074b998ddf4c8a1c30ea84" } modify_external_catalogs: true
I20250629 01:52:04.125777 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0a60f2cb55074b998ddf4c8a1c30ea84" } modify_external_catalogs: true
I20250629 01:52:04.136942 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:04.137149 450 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet 76118a6e684448b09f244d44c909ccaa (table default.Table_5794ea9989b54ab9bd7ecca30428c971 [id=dcc07a46945a4add8e667bfea51dfed7]). Expected version 5 got 0
I20250629 01:52:04.139374 2760 raft_consensus.cc:513] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.140038 2760 leader_election.cc:290] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.141449 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet fb03913296df45188e23337042e96d23
I20250629 01:52:04.143512 744 tablet_service.cc:1515] Processing DeleteTablet for tablet fb03913296df45188e23337042e96d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.144165 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet fb03913296df45188e23337042e96d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.145316 892 tablet_service.cc:1515] Processing DeleteTablet for tablet fb03913296df45188e23337042e96d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.145838 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1430) took 2112 ms. Trace:
I20250629 01:52:04.145915 1156 rpcz_store.cc:276] 0629 01:52:02.032892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.032960 (+ 68us) service_pool.cc:225] Handling call
0629 01:52:03.868660 (+1835700us) ts_tablet_manager.cc:1145] Deleting tablet b6fa908ded8240bb962674e1efb29c74
0629 01:52:03.868668 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:03.950013 (+ 81345us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.040993 (+ 90980us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.145820 (+104827us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1835249}
I20250629 01:52:04.146191 1156 tablet_replica.cc:331] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:04.146345 1156 raft_consensus.cc:2241] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:04.146466 1156 raft_consensus.cc:2270] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.146518 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3b55c66136ba43d7b535750e04b72308
I20250629 01:52:04.146931 1156 ts_tablet_manager.cc:1905] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.147845 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet b6fa908ded8240bb962674e1efb29c74 (table default.Table_2ba17dbba43040089a9c2d400550f15f [id=9e20604c9f594334b255b4ca0a36012f]) successfully deleted
I20250629 01:52:04.148170 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b55c66136ba43d7b535750e04b72308 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.148500 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b55c66136ba43d7b535750e04b72308 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.148963 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 3b55c66136ba43d7b535750e04b72308 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.149055 2859 ts_tablet_manager.cc:1918] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.149137 2859 log.cc:1199] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/4d9ef77181ba48f3a6de16b945c8df1a
I20250629 01:52:04.149471 2859 ts_tablet_manager.cc:1939] T 4d9ef77181ba48f3a6de16b945c8df1a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:04.152446 2896 raft_consensus.cc:491] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.152556 2896 raft_consensus.cc:513] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.153052 2896 leader_election.cc:290] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:04.153947 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5771fdd618e45118df2b6dab7f668d9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:04.154361 988 leader_election.cc:343] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.155516 776 raft_consensus.cc:2466] T 9ed1ee73bb8f4761806a66f988b57bf9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:04.155840 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 9e53a3ecef1c4954a0f4a48c7608a5d3
I20250629 01:52:04.157140 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.157299 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.157634 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e53a3ecef1c4954a0f4a48c7608a5d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:04.158164 850 leader_election.cc:343] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 7908afa2abaa4521b44f954e0790f986
I20250629 01:52:04.158689 916 raft_consensus.cc:3058] T 6a9d2aa33a5c4d9ebecaa83c32b39dcb P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.163606 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5771fdd618e45118df2b6dab7f668d9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:04.163734 776 raft_consensus.cc:3058] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.164208 986 leader_election.cc:304] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.164525 2760 raft_consensus.cc:2802] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.164631 2760 raft_consensus.cc:695] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:04.164817 2760 consensus_queue.cc:237] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:04.172263 850 leader_election.cc:343] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 7908afa2abaa4521b44f954e0790f986
I20250629 01:52:04.172390 850 leader_election.cc:304] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.172647 2896 raft_consensus.cc:2747] T 7908afa2abaa4521b44f954e0790f986 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.173310 1158 ts_tablet_manager.cc:1918] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.173434 1158 log.cc:1199] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/4c7b76cb56b24604b13ee29c7f576ce2
I20250629 01:52:04.173740 1158 ts_tablet_manager.cc:1939] T 4c7b76cb56b24604b13ee29c7f576ce2 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:04.177644 1013 tablet_service.cc:1468] Processing CreateTablet for tablet e36e7647ca3a4f99bc602a5e2dafffd5 (DEFAULT_TABLE table=default.Table_c65f50be17b4466a9991d2195fef4c5f [id=f7f3250970ec4d218d5d7340d8acb189]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.177969 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36e7647ca3a4f99bc602a5e2dafffd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.183094 1154 tablet_bootstrap.cc:700] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913: Time spent opening tablet: real 0.124s user 0.000s sys 0.000s
I20250629 01:52:04.183305 1154 tablet_bootstrap.cc:654] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.184921 744 tablet_service.cc:1468] Processing CreateTablet for tablet e36e7647ca3a4f99bc602a5e2dafffd5 (DEFAULT_TABLE table=default.Table_c65f50be17b4466a9991d2195fef4c5f [id=f7f3250970ec4d218d5d7340d8acb189]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.186185 879 tablet_service.cc:1468] Processing CreateTablet for tablet 5fbf8715afde447696dc1ac530739282 (DEFAULT_TABLE table=default.Table_60a30fc239a64f36965d6eb696944710 [id=82007d08d15845c39672a2be1583976c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.186638 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fbf8715afde447696dc1ac530739282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.198042 2907 raft_consensus.cc:491] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.198143 2907 raft_consensus.cc:513] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.198156 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 5fbf8715afde447696dc1ac530739282 (DEFAULT_TABLE table=default.Table_60a30fc239a64f36965d6eb696944710 [id=82007d08d15845c39672a2be1583976c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.198423 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fbf8715afde447696dc1ac530739282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.198599 2907 leader_election.cc:290] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.199298 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb98aa9d365541969ea1dc70960760b2" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.199801 710 leader_election.cc:343] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: bb98aa9d365541969ea1dc70960760b2
W20250629 01:52:04.200047 712 leader_election.cc:343] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.200181 712 leader_election.cc:304] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:04.200392 2907 raft_consensus.cc:2747] T bb98aa9d365541969ea1dc70960760b2 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.202409 740 tablet_service.cc:1468] Processing CreateTablet for tablet 5fbf8715afde447696dc1ac530739282 (DEFAULT_TABLE table=default.Table_60a30fc239a64f36965d6eb696944710 [id=82007d08d15845c39672a2be1583976c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.203510 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9814bb0688d34f939aca5b27c0c27584 (DEFAULT_TABLE table=default.Table_99304b2c156d4784ba9cb3aed7494313 [id=5497e304ac764b41add4b43ff3ca35b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.203812 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9814bb0688d34f939aca5b27c0c27584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.206755 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 9814bb0688d34f939aca5b27c0c27584 (DEFAULT_TABLE table=default.Table_99304b2c156d4784ba9cb3aed7494313 [id=5497e304ac764b41add4b43ff3ca35b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.207038 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9814bb0688d34f939aca5b27c0c27584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.210103 750 tablet_service.cc:1468] Processing CreateTablet for tablet 9814bb0688d34f939aca5b27c0c27584 (DEFAULT_TABLE table=default.Table_99304b2c156d4784ba9cb3aed7494313 [id=5497e304ac764b41add4b43ff3ca35b4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:04.212399 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 378878us
I20250629 01:52:04.221203 745 tablet_service.cc:1468] Processing CreateTablet for tablet 1e7332a9109e470c9d097745c7490e2f (DEFAULT_TABLE table=default.Table_0fcaee65bbdc47778737eac4a8c77fd5 [id=877ec694432c4512bfe8faf86f0699b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.224324 884 tablet_service.cc:1468] Processing CreateTablet for tablet 1e7332a9109e470c9d097745c7490e2f (DEFAULT_TABLE table=default.Table_0fcaee65bbdc47778737eac4a8c77fd5 [id=877ec694432c4512bfe8faf86f0699b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.224624 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7332a9109e470c9d097745c7490e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.226503 880 tablet_service.cc:1468] Processing CreateTablet for tablet b65cb1617fb4444796cf10814aff64fd (DEFAULT_TABLE table=default.Table_a24244a6087d4f0d98c841951fc3bb1a [id=73433bfab3244139b331ecf5220609ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.226871 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b65cb1617fb4444796cf10814aff64fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.227052 1017 tablet_service.cc:1468] Processing CreateTablet for tablet b65cb1617fb4444796cf10814aff64fd (DEFAULT_TABLE table=default.Table_a24244a6087d4f0d98c841951fc3bb1a [id=73433bfab3244139b331ecf5220609ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.227325 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b65cb1617fb4444796cf10814aff64fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.227490 751 tablet_service.cc:1468] Processing CreateTablet for tablet b65cb1617fb4444796cf10814aff64fd (DEFAULT_TABLE table=default.Table_a24244a6087d4f0d98c841951fc3bb1a [id=73433bfab3244139b331ecf5220609ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.227648 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 1e7332a9109e470c9d097745c7490e2f (DEFAULT_TABLE table=default.Table_0fcaee65bbdc47778737eac4a8c77fd5 [id=877ec694432c4512bfe8faf86f0699b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.227921 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7332a9109e470c9d097745c7490e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.228412 888 tablet_service.cc:1468] Processing CreateTablet for tablet a70d4725e3f54ac2a7397a625516e651 (DEFAULT_TABLE table=default.Table_19a73abe17d0485088732b0719e69baa [id=b39efdf4a62e4fa2ade433e1790271a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.228715 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d4725e3f54ac2a7397a625516e651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.230943 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a70d4725e3f54ac2a7397a625516e651 (DEFAULT_TABLE table=default.Table_19a73abe17d0485088732b0719e69baa [id=b39efdf4a62e4fa2ade433e1790271a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.231263 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d4725e3f54ac2a7397a625516e651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.232280 885 tablet_service.cc:1468] Processing CreateTablet for tablet b384d1e66e9c420480bd308b8748638c (DEFAULT_TABLE table=default.Table_da80d6865112454b8f206b478bbda7cc [id=ffe93c25c2d84c958a5a6d6a7a5a1a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.232600 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b384d1e66e9c420480bd308b8748638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.236166 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1378) took 1294 ms. Trace:
I20250629 01:52:04.236236 2859 rpcz_store.cc:276] 0629 01:52:02.941883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.941920 (+ 37us) service_pool.cc:225] Handling call
0629 01:52:03.968401 (+1026481us) ts_tablet_manager.cc:1145] Deleting tablet 4d9ef77181ba48f3a6de16b945c8df1a
0629 01:52:03.968409 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.038274 (+ 69865us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.148964 (+110690us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.236152 (+ 87188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1026369}
I20250629 01:52:04.236495 2859 tablet_replica.cc:331] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:04.236605 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 4d9ef77181ba48f3a6de16b945c8df1a (table default.Table_f2175d9af5064903ab1c97c2035e2527 [id=207fae1412fa4f1da7eb67dadd6f986e]) successfully deleted
I20250629 01:52:04.236646 2859 raft_consensus.cc:2241] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:04.236759 2859 raft_consensus.cc:2270] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.237215 2859 ts_tablet_manager.cc:1905] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.239941 895 tablet_service.cc:1468] Processing CreateTablet for tablet 30fc706070da4952bc89b31bc5f877a1 (DEFAULT_TABLE table=default.Table_a205f56d1700435b9739b3651efae052 [id=8ce0fe9e8b0f46fcbcaa622b9652c1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.240549 2894 raft_consensus.cc:513] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.241052 2894 leader_election.cc:290] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.241437 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ca02bcd9467496c9057b3de9c1b1725" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:04.241804 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 30fc706070da4952bc89b31bc5f877a1 (DEFAULT_TABLE table=default.Table_a205f56d1700435b9739b3651efae052 [id=8ce0fe9e8b0f46fcbcaa622b9652c1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.242069 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30fc706070da4952bc89b31bc5f877a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.242614 757 tablet_service.cc:1468] Processing CreateTablet for tablet b384d1e66e9c420480bd308b8748638c (DEFAULT_TABLE table=default.Table_da80d6865112454b8f206b478bbda7cc [id=ffe93c25c2d84c958a5a6d6a7a5a1a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.243012 753 tablet_service.cc:1468] Processing CreateTablet for tablet 30fc706070da4952bc89b31bc5f877a1 (DEFAULT_TABLE table=default.Table_a205f56d1700435b9739b3651efae052 [id=8ce0fe9e8b0f46fcbcaa622b9652c1cc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:04.243558 988 leader_election.cc:343] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.244282 1021 tablet_service.cc:1468] Processing CreateTablet for tablet b384d1e66e9c420480bd308b8748638c (DEFAULT_TABLE table=default.Table_da80d6865112454b8f206b478bbda7cc [id=ffe93c25c2d84c958a5a6d6a7a5a1a2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.244546 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b384d1e66e9c420480bd308b8748638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.245956 776 raft_consensus.cc:2466] T c5771fdd618e45118df2b6dab7f668d9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:04.249228 896 tablet_service.cc:1468] Processing CreateTablet for tablet 2bd218fc4ab94adb8a1680e84bb88c00 (DEFAULT_TABLE table=default.Table_b0da5a36b5c141cc848f23ee43a30fa1 [id=1b2e5bc17a524432af71014b105fd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.257747 2894 raft_consensus.cc:491] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.257867 2894 raft_consensus.cc:513] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.258348 2894 leader_election.cc:290] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.258824 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2045012c80d049eaabf7dec1df28b087" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.259297 988 leader_election.cc:343] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.262825 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ca02bcd9467496c9057b3de9c1b1725" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:04.263959 986 leader_election.cc:304] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.264356 2894 raft_consensus.cc:2802] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.264457 2894 raft_consensus.cc:695] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:04.264653 2894 consensus_queue.cc:237] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.264932 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_da8ce3889e724c1291c4dc0a46f206f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:04.270658 2935 tablet.cc:1722] T 3da0deb865984f70913323364ca38703 P 4712643b602e4f898752fcab6a1e5913: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:04.271853 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0d7a6dc928ef432996bb10611192b6a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.275521 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d7a6dc928ef432996bb10611192b6a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.277670 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_63515e00ce034ce3af0c189f3ec838fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.278103 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63515e00ce034ce3af0c189f3ec838fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.279174 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_df7ca1b23ac246b2968ae5b317a47fb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.279489 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df7ca1b23ac246b2968ae5b317a47fb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.279820 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_0b7e1ffded594ba6aedda2a5482fa168"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.280189 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b7e1ffded594ba6aedda2a5482fa168 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.283566 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_121b994e3a964726b696940dae054ccd" } new_table_name: "default.Table_9e64bce37317401aa8f3c511231154be" modify_external_catalogs: true
I20250629 01:52:04.284699 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_69903da7c96841cc8b8e2b2f9fb281d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.285188 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69903da7c96841cc8b8e2b2f9fb281d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.287717 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c950187cddb24b5eabf5f6d04516faaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:04.287846 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7c24c1ccd9e64a1d9c9b9cc86ff93ca6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.288132 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c950187cddb24b5eabf5f6d04516faaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:04.288300 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c24c1ccd9e64a1d9c9b9cc86ff93ca6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.294560 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_095e7cfff96a42deb7f4800b2d974ee6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.294987 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_095e7cfff96a42deb7f4800b2d974ee6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:04.295830 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da8ce3889e724c1291c4dc0a46f206f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.296209 2907 raft_consensus.cc:491] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.296535 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e2d0b8bcad624e1ca0dcf18b3cc5b3ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.296953 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2d0b8bcad624e1ca0dcf18b3cc5b3ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.301090 1159 tablet_bootstrap.cc:492] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.301263 1159 ts_tablet_manager.cc:1397] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.348s user 0.025s sys 0.051s
I20250629 01:52:04.301227 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_6f957d68e3dd4ba9a5bf56cc925385bc" } new_table_name: "default.Table_f08b2eabe99142c7a5edcfd416f3fa1d" modify_external_catalogs: true
I20250629 01:52:04.302387 1159 raft_consensus.cc:357] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.302594 1159 raft_consensus.cc:383] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.302702 1159 raft_consensus.cc:738] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.302976 1159 consensus_queue.cc:260] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.303624 1159 ts_tablet_manager.cc:1428] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250629 01:52:04.310492 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18a5af7b1c8b40dd8751d5d7a694c3eb consensus state has no leader
I20250629 01:52:04.319471 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9c7f015311024589a1da970cdda15955" } new_table_name: "default.Table_2cde373b9ede4945a8e4347a43d140a1" modify_external_catalogs: true
I20250629 01:52:04.320636 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1bcb2d8c2d424b15834889cd98119d72" } modify_external_catalogs: true
I20250629 01:52:04.320755 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1bcb2d8c2d424b15834889cd98119d72" } modify_external_catalogs: true
I20250629 01:52:04.321017 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5fba3653ae2c4de78346d2983120e610" } new_table_name: "default.Table_13bb6581c5d14f3eb135c6f7bb9c1b96" modify_external_catalogs: true
I20250629 01:52:04.322417 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_073d8d6803d541acb4e3549a8ca2dca2" } new_table_name: "default.Table_03bc9727410a47acabbcdf4a0a7cfcfb" modify_external_catalogs: true
I20250629 01:52:04.324896 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2bd218fc4ab94adb8a1680e84bb88c00 (DEFAULT_TABLE table=default.Table_b0da5a36b5c141cc848f23ee43a30fa1 [id=1b2e5bc17a524432af71014b105fd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.325891 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1d228257fd614e23b098ddd7c3e7d0d8" } modify_external_catalogs: true
I20250629 01:52:04.326026 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_1d228257fd614e23b098ddd7c3e7d0d8" } modify_external_catalogs: true
I20250629 01:52:04.329571 892 tablet_service.cc:1468] Processing CreateTablet for tablet 7b8352b11c4c4fddbe332d794d538495 (DEFAULT_TABLE table=default.Table_122278fe170445189b568ab13655f5f6 [id=ab5b44ac46da419a854ed479ecfd30e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.331219 1154 tablet_bootstrap.cc:492] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.331316 1154 ts_tablet_manager.cc:1397] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.501s user 0.001s sys 0.001s
I20250629 01:52:04.331959 1154 raft_consensus.cc:357] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.333314 742 tablet_service.cc:1468] Processing CreateTablet for tablet a70d4725e3f54ac2a7397a625516e651 (DEFAULT_TABLE table=default.Table_19a73abe17d0485088732b0719e69baa [id=b39efdf4a62e4fa2ade433e1790271a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.333626 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e36e7647ca3a4f99bc602a5e2dafffd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.333909 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b65cb1617fb4444796cf10814aff64fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.333961 756 tablet_service.cc:1468] Processing CreateTablet for tablet 2bd218fc4ab94adb8a1680e84bb88c00 (DEFAULT_TABLE table=default.Table_b0da5a36b5c141cc848f23ee43a30fa1 [id=1b2e5bc17a524432af71014b105fd1a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.334252 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd218fc4ab94adb8a1680e84bb88c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.335270 1052 raft_consensus.cc:2466] T bccdcd9de08648bb925db6fe56f97d20 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:04.337208 710 leader_election.cc:304] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.337503 2651 consensus_queue.cc:1035] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:04.337860 2651 raft_consensus.cc:2802] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.337967 2651 raft_consensus.cc:695] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:04.338168 2651 consensus_queue.cc:237] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.339226 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2045012c80d049eaabf7dec1df28b087" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:04.339563 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30fc706070da4952bc89b31bc5f877a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.339996 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd218fc4ab94adb8a1680e84bb88c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.340206 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fbf8715afde447696dc1ac530739282. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.344223 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b384d1e66e9c420480bd308b8748638c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.344527 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70d4725e3f54ac2a7397a625516e651. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.348197 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9814bb0688d34f939aca5b27c0c27584. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.352048 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bd218fc4ab94adb8a1680e84bb88c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.352070 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30fc706070da4952bc89b31bc5f877a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.352427 1158 tablet_replica.cc:331] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:04.352609 1158 raft_consensus.cc:2241] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:04.352725 1158 raft_consensus.cc:2270] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.353250 1158 ts_tablet_manager.cc:1905] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.357661 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 4c7b76cb56b24604b13ee29c7f576ce2 (table default.Table_af7702628dd2458d831dcc0ce1be9327 [id=ae19e8ff3bdb4232969c0b35da790b4d]) successfully deleted
I20250629 01:52:04.359987 2548 consensus_queue.cc:1035] T 8635903caac9480184c32e6689870485 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:04.364809 2908 tablet.cc:1722] T 3da0deb865984f70913323364ca38703 P ef5c5979bb9e475f983af7c3e9e0e019: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:04.365235 2907 raft_consensus.cc:513] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.365764 2907 leader_election.cc:290] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.365926 772 raft_consensus.cc:3058] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.368724 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1ebbcb2c26b4667b2405154a9d8fccb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.368911 1044 raft_consensus.cc:2466] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:04.369370 710 leader_election.cc:304] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:04.369822 2907 raft_consensus.cc:2802] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.369906 2907 raft_consensus.cc:491] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.369976 2907 raft_consensus.cc:3058] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:04.370386 712 consensus_peers.cc:489] T 2ec92eddfabf40918a6cabaff5fe113a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:04.370486 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 62265.3us
I20250629 01:52:04.377615 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1ebbcb2c26b4667b2405154a9d8fccb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.381225 988 consensus_peers.cc:489] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:04.381758 1018 tablet_service.cc:1468] Processing CreateTablet for tablet 7b8352b11c4c4fddbe332d794d538495 (DEFAULT_TABLE table=default.Table_122278fe170445189b568ab13655f5f6 [id=ab5b44ac46da419a854ed479ecfd30e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.383782 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7b8352b11c4c4fddbe332d794d538495 (DEFAULT_TABLE table=default.Table_122278fe170445189b568ab13655f5f6 [id=ab5b44ac46da419a854ed479ecfd30e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:04.385205 986 leader_election.cc:343] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.385337 986 leader_election.cc:304] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.385552 2898 raft_consensus.cc:2747] T 2045012c80d049eaabf7dec1df28b087 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.386343 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 652d8fddc3bb45d9ad82d2b20976e06a (DEFAULT_TABLE table=default.Table_65aa0434a5d7401aa27aeaf83d5e1ecf [id=2d6999b0dc144acc90df2313f31a2f37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.386802 880 tablet_service.cc:1468] Processing CreateTablet for tablet 652d8fddc3bb45d9ad82d2b20976e06a (DEFAULT_TABLE table=default.Table_65aa0434a5d7401aa27aeaf83d5e1ecf [id=2d6999b0dc144acc90df2313f31a2f37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.388041 2548 raft_consensus.cc:491] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.388175 2548 raft_consensus.cc:513] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.388655 2548 leader_election.cc:290] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.388986 743 tablet_service.cc:1468] Processing CreateTablet for tablet 652d8fddc3bb45d9ad82d2b20976e06a (DEFAULT_TABLE table=default.Table_65aa0434a5d7401aa27aeaf83d5e1ecf [id=2d6999b0dc144acc90df2313f31a2f37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.389568 896 tablet_service.cc:1468] Processing CreateTablet for tablet cdfdfe8e0f4b41c5972c3d7656bc61e1 (DEFAULT_TABLE table=default.Table_ae1722f51df94fb390a076f2139a7e9c [id=ac9c8af9084f4ed29c79e638d32cf012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.389863 1023 tablet_service.cc:1468] Processing CreateTablet for tablet cdfdfe8e0f4b41c5972c3d7656bc61e1 (DEFAULT_TABLE table=default.Table_ae1722f51df94fb390a076f2139a7e9c [id=ac9c8af9084f4ed29c79e638d32cf012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.391747 742 tablet_service.cc:1468] Processing CreateTablet for tablet cdfdfe8e0f4b41c5972c3d7656bc61e1 (DEFAULT_TABLE table=default.Table_ae1722f51df94fb390a076f2139a7e9c [id=ac9c8af9084f4ed29c79e638d32cf012]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.392863 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01d95356c7a7419bb56ddf67d451eadc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.393034 1044 raft_consensus.cc:2466] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:04.393350 710 leader_election.cc:304] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:04.393600 2548 raft_consensus.cc:2802] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.393680 2548 raft_consensus.cc:491] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.393752 2548 raft_consensus.cc:3058] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.395155 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01d95356c7a7419bb56ddf67d451eadc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:04.395593 2894 consensus_queue.cc:1035] T c62678645ff54aa6b3809212f4d66ec2 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:52:04.395761 712 leader_election.cc:343] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:04.396044 712 leader_election.cc:343] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.408783 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7332a9109e470c9d097745c7490e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.409020 895 tablet_service.cc:1468] Processing CreateTablet for tablet 69464ae3512b47c29c9022c78efeef39 (DEFAULT_TABLE table=default.Table_d0563326136c457aabee62f8d46d730c [id=b899f55108c74f1d88dca81458b38dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.411789 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 69464ae3512b47c29c9022c78efeef39 (DEFAULT_TABLE table=default.Table_d0563326136c457aabee62f8d46d730c [id=b899f55108c74f1d88dca81458b38dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.413249 748 tablet_service.cc:1468] Processing CreateTablet for tablet 69464ae3512b47c29c9022c78efeef39 (DEFAULT_TABLE table=default.Table_d0563326136c457aabee62f8d46d730c [id=b899f55108c74f1d88dca81458b38dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.415700 1159 tablet_bootstrap.cc:492] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.415746 885 tablet_service.cc:1468] Processing CreateTablet for tablet 96ec2df356e046e0bec740eed295fe00 (DEFAULT_TABLE table=default.Table_eb68b194087045e2886841418bd9c3ed [id=1e9e3c237c61426e9365b7cc6acf22ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.416504 1154 raft_consensus.cc:383] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.419350 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ad1ba1fec9bf49a5a44a193802edde0a
I20250629 01:52:04.421300 1159 tablet_bootstrap.cc:654] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.421588 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0b7e1ffded594ba6aedda2a5482fa168" } modify_external_catalogs: true
I20250629 01:52:04.421756 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_0b7e1ffded594ba6aedda2a5482fa168" } modify_external_catalogs: true
I20250629 01:52:04.422147 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6" } modify_external_catalogs: true
I20250629 01:52:04.422288 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d3f9bfb15f7f49ab8c93718efaf58ac6" } modify_external_catalogs: true
I20250629 01:52:04.423226 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1ba1fec9bf49a5a44a193802edde0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.426890 752 tablet_service.cc:1468] Processing CreateTablet for tablet 96ec2df356e046e0bec740eed295fe00 (DEFAULT_TABLE table=default.Table_eb68b194087045e2886841418bd9c3ed [id=1e9e3c237c61426e9365b7cc6acf22ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.429029 747 tablet_service.cc:1468] Processing CreateTablet for tablet a69afb420f4d491f811e22da1a5014a5 (DEFAULT_TABLE table=default.Table_0198d8f0bc1443ac86dbfa3f74614a93 [id=f2b029e30df54ddcaf9a3605506a552a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.430989 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_45ad1da33f23408c858cd346e7ac71bf" } modify_external_catalogs: true
I20250629 01:52:04.431660 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_45ad1da33f23408c858cd346e7ac71bf" } modify_external_catalogs: true
I20250629 01:52:04.433256 738 tablet_service.cc:1468] Processing CreateTablet for tablet 5a56bef21caf4009ae9f4360ec66ca63 (DEFAULT_TABLE table=default.Table_73d48f855413440183cc015c9e93636c [id=a1df354f018f4ff5a5ece9c21de54705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.433626 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a56bef21caf4009ae9f4360ec66ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.434229 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8352b11c4c4fddbe332d794d538495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.433252 1156 ts_tablet_manager.cc:1918] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.434788 1156 log.cc:1199] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/dc841eb171d04b118b1e705ceed4c861
I20250629 01:52:04.435111 1156 ts_tablet_manager.cc:1939] T dc841eb171d04b118b1e705ceed4c861 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:04.435606 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69464ae3512b47c29c9022c78efeef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.436007 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ec2df356e046e0bec740eed295fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.436213 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69afb420f4d491f811e22da1a5014a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.436807 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652d8fddc3bb45d9ad82d2b20976e06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.437273 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdfdfe8e0f4b41c5972c3d7656bc61e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.440867 888 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1ba1fec9bf49a5a44a193802edde0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.440896 884 tablet_service.cc:1468] Processing CreateTablet for tablet a69afb420f4d491f811e22da1a5014a5 (DEFAULT_TABLE table=default.Table_0198d8f0bc1443ac86dbfa3f74614a93 [id=f2b029e30df54ddcaf9a3605506a552a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.441286 879 tablet_service.cc:1468] Processing CreateTablet for tablet 5a56bef21caf4009ae9f4360ec66ca63 (DEFAULT_TABLE table=default.Table_73d48f855413440183cc015c9e93636c [id=a1df354f018f4ff5a5ece9c21de54705]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.443924 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 96ec2df356e046e0bec740eed295fe00 (DEFAULT_TABLE table=default.Table_eb68b194087045e2886841418bd9c3ed [id=1e9e3c237c61426e9365b7cc6acf22ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.444485 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet ad1ba1fec9bf49a5a44a193802edde0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.445508 1021 tablet_service.cc:1468] Processing CreateTablet for tablet a69afb420f4d491f811e22da1a5014a5 (DEFAULT_TABLE table=default.Table_0198d8f0bc1443ac86dbfa3f74614a93 [id=f2b029e30df54ddcaf9a3605506a552a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:04.445837 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 5a56bef21caf4009ae9f4360ec66ca63 (DEFAULT_TABLE table=default.Table_73d48f855413440183cc015c9e93636c [id=a1df354f018f4ff5a5ece9c21de54705]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:04.447146 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77d497c0dc4f49caa22758d3d7517a95 consensus state has no leader
I20250629 01:52:04.447517 1154 raft_consensus.cc:738] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
W20250629 01:52:04.447785 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 789314589f284283a0d37831f5651d3f consensus state has no leader
I20250629 01:52:04.451979 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652d8fddc3bb45d9ad82d2b20976e06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.452210 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8352b11c4c4fddbe332d794d538495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.452396 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdfdfe8e0f4b41c5972c3d7656bc61e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.452514 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ec2df356e046e0bec740eed295fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.452864 2859 ts_tablet_manager.cc:1918] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.452996 2859 log.cc:1199] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/06e31520d3ad4c3395a59049755a4725
I20250629 01:52:04.453297 2859 ts_tablet_manager.cc:1939] T 06e31520d3ad4c3395a59049755a4725 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:04.453439 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69464ae3512b47c29c9022c78efeef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.453977 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a56bef21caf4009ae9f4360ec66ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.454319 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69afb420f4d491f811e22da1a5014a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.464648 1154 consensus_queue.cc:260] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.465662 1154 ts_tablet_manager.cc:1428] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.134s user 0.001s sys 0.001s
I20250629 01:52:04.465951 1154 tablet_bootstrap.cc:492] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.469691 1154 tablet_bootstrap.cc:654] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.481870 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 4aa8dfd712c043ad9f8ff56b064bb8fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.482295 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b8352b11c4c4fddbe332d794d538495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.482514 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdfdfe8e0f4b41c5972c3d7656bc61e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.482697 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 652d8fddc3bb45d9ad82d2b20976e06a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.482833 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69464ae3512b47c29c9022c78efeef39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.483116 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96ec2df356e046e0bec740eed295fe00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.483495 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69afb420f4d491f811e22da1a5014a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.483902 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a56bef21caf4009ae9f4360ec66ca63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:04.493821 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d1ce1ecb55ae42c39c9835dc139076b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.494318 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1ce1ecb55ae42c39c9835dc139076b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.495532 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_b77011fde88a45f5b349d797b437ac27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.495906 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b77011fde88a45f5b349d797b437ac27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.496485 772 raft_consensus.cc:2466] T 6ca02bcd9467496c9057b3de9c1b1725 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:04.510351 2898 raft_consensus.cc:491] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.510483 2898 raft_consensus.cc:513] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.511108 2898 leader_election.cc:290] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.512777 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_9c37a49e7730465f9c278760fba0c5dd" } new_table_name: "default.Table_761daa0437d34b20900555e0b398a26c" modify_external_catalogs: true
I20250629 01:52:04.514084 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e4e2cd9a49bf4b35885c1de3c2fae5e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.514525 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4e2cd9a49bf4b35885c1de3c2fae5e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.519241 2969 raft_consensus.cc:491] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.528520 2969 raft_consensus.cc:513] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.529170 2969 leader_election.cc:290] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.531355 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557ecee0d6e84f77ad10bbdd27859ee3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.531888 988 leader_election.cc:343] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.541350 2907 raft_consensus.cc:513] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.541867 2907 leader_election.cc:290] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.542439 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1376) took 1600 ms. Trace:
I20250629 01:52:04.542606 2859 rpcz_store.cc:276] 0629 01:52:02.941695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.942028 (+ 333us) service_pool.cc:225] Handling call
0629 01:52:04.236458 (+1294430us) ts_tablet_manager.cc:1145] Deleting tablet 06e31520d3ad4c3395a59049755a4725
0629 01:52:04.236467 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.336625 (+100158us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.452789 (+116164us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.542423 (+ 89634us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294326}
I20250629 01:52:04.543227 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1369) took 1602 ms. Trace:
I20250629 01:52:04.543341 2859 rpcz_store.cc:276] 0629 01:52:02.940880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.941450 (+ 570us) service_pool.cc:225] Handling call
0629 01:52:04.543070 (+1601620us) ts_tablet_manager.cc:1145] Deleting tablet 72b2caf969db48458ea007ae03a0640f
0629 01:52:04.543078 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.543212 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600110}
I20250629 01:52:04.541867 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1436) took 2404 ms. Trace:
I20250629 01:52:04.543802 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1393) took 1475 ms. Trace:
I20250629 01:52:04.543725 1156 rpcz_store.cc:276] 0629 01:52:02.137225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.137268 (+ 43us) service_pool.cc:225] Handling call
0629 01:52:04.146149 (+2008881us) ts_tablet_manager.cc:1145] Deleting tablet dc841eb171d04b118b1e705ceed4c861
0629 01:52:04.146157 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.234532 (+ 88375us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.433161 (+198629us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.541849 (+108688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008713}
I20250629 01:52:04.543853 2859 rpcz_store.cc:276] 0629 01:52:03.068605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.068745 (+ 140us) service_pool.cc:225] Handling call
0629 01:52:04.543691 (+1474946us) ts_tablet_manager.cc:1145] Deleting tablet 12873ec9a5574e789f9d669ddda85704
0629 01:52:04.543698 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.543779 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474810}
I20250629 01:52:04.543972 1156 tablet_replica.cc:331] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:04.544147 1156 raft_consensus.cc:2241] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:04.544210 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1394) took 1475 ms. Trace:
I20250629 01:52:04.544258 2859 rpcz_store.cc:276] 0629 01:52:03.068998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.069028 (+ 30us) service_pool.cc:225] Handling call
0629 01:52:04.544120 (+1475092us) ts_tablet_manager.cc:1145] Deleting tablet 4416bbec7d1c40df9a835b503d368d4c
0629 01:52:04.544127 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.544197 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474972}
I20250629 01:52:04.544390 1156 pending_rounds.cc:70] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:04.544432 1156 pending_rounds.cc:77] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759232067801088 op_type: NO_OP noop_request { }
I20250629 01:52:04.544502 1156 raft_consensus.cc:2887] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:04.544569 1156 raft_consensus.cc:2270] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.544646 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1392) took 1476 ms. Trace:
I20250629 01:52:04.544697 2859 rpcz_store.cc:276] 0629 01:52:03.068543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.072133 (+ 3590us) service_pool.cc:225] Handling call
0629 01:52:04.544515 (+1472382us) ts_tablet_manager.cc:1145] Deleting tablet 1ac60a024a074953b513ae4f60840d25
0629 01:52:04.544523 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.544632 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472207}
I20250629 01:52:04.545040 1156 ts_tablet_manager.cc:1905] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.545051 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1395) took 1412 ms. Trace:
I20250629 01:52:04.545101 2859 rpcz_store.cc:276] 0629 01:52:03.132467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.136369 (+ 3902us) service_pool.cc:225] Handling call
0629 01:52:04.544969 (+1408600us) ts_tablet_manager.cc:1145] Deleting tablet 13d779335d09442682554322b59f915a
0629 01:52:04.544975 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.545039 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408476}
I20250629 01:52:04.545385 2859 tablet_replica.cc:331] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:04.545533 2859 raft_consensus.cc:2241] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:04.545655 2859 raft_consensus.cc:2270] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.546320 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 06e31520d3ad4c3395a59049755a4725 (table default.Table_2e69d514f50649f79824a1cb1da60444 [id=d843c94cf5f847249d82ea7944980d89]) successfully deleted
W20250629 01:52:04.546492 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 72b2caf969db48458ea007ae03a0640f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72b2caf969db48458ea007ae03a0640f already in progress: creating tablet
W20250629 01:52:04.546656 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 12873ec9a5574e789f9d669ddda85704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12873ec9a5574e789f9d669ddda85704 already in progress: creating tablet
W20250629 01:52:04.546770 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4416bbec7d1c40df9a835b503d368d4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4416bbec7d1c40df9a835b503d368d4c already in progress: creating tablet
W20250629 01:52:04.546898 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 1ac60a024a074953b513ae4f60840d25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac60a024a074953b513ae4f60840d25 already in progress: creating tablet
W20250629 01:52:04.547027 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 13d779335d09442682554322b59f915a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13d779335d09442682554322b59f915a already in progress: creating tablet
I20250629 01:52:04.546059 2859 ts_tablet_manager.cc:1905] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.581945 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557ecee0d6e84f77ad10bbdd27859ee3" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:04.582809 986 leader_election.cc:304] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.583127 2898 raft_consensus.cc:2802] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.583227 2898 raft_consensus.cc:695] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:04.583431 2898 consensus_queue.cc:237] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.585655 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1ebbcb2c26b4667b2405154a9d8fccb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:04.585698 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67e90a48af8f48bba3b3fd4f4110d367" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.586686 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 106351us
I20250629 01:52:04.587358 2760 consensus_queue.cc:1035] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250629 01:52:04.588068 986 leader_election.cc:343] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.588236 986 leader_election.cc:304] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.588449 2760 raft_consensus.cc:2747] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.589952 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "67e90a48af8f48bba3b3fd4f4110d367" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.590535 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1ebbcb2c26b4667b2405154a9d8fccb" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:04.590672 1048 raft_consensus.cc:3058] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.591322 1154 tablet_bootstrap.cc:492] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.591466 1154 ts_tablet_manager.cc:1397] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.126s user 0.002s sys 0.002s
I20250629 01:52:04.592136 1154 raft_consensus.cc:357] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.592319 1154 raft_consensus.cc:383] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.592393 1154 raft_consensus.cc:738] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.592715 1154 consensus_queue.cc:260] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.593794 1154 ts_tablet_manager.cc:1428] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:04.594337 1154 tablet_bootstrap.cc:492] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:52:04.599363 712 leader_election.cc:343] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:04.599606 712 leader_election.cc:343] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.602538 1154 tablet_bootstrap.cc:654] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:04.613866 710 leader_election.cc:343] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Illegal state: Tablet not RUNNING: STOPPED
I20250629 01:52:04.614114 710 leader_election.cc:304] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, 4712643b602e4f898752fcab6a1e5913
I20250629 01:52:04.614574 2964 raft_consensus.cc:2747] T 67e90a48af8f48bba3b3fd4f4110d367 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:04.631670 988 consensus_peers.cc:489] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:04.631986 1048 raft_consensus.cc:2466] T b1ebbcb2c26b4667b2405154a9d8fccb P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:04.635139 2548 raft_consensus.cc:513] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.635690 2548 leader_election.cc:290] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.636061 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01d95356c7a7419bb56ddf67d451eadc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:04.636229 1158 ts_tablet_manager.cc:1918] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.636312 1158 log.cc:1199] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/67e90a48af8f48bba3b3fd4f4110d367
I20250629 01:52:04.636562 1158 ts_tablet_manager.cc:1939] T 67e90a48af8f48bba3b3fd4f4110d367 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:52:04.636622 712 leader_election.cc:343] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.640880 1154 tablet_bootstrap.cc:492] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.641036 1154 ts_tablet_manager.cc:1397] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250629 01:52:04.641645 1154 raft_consensus.cc:357] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.641834 1154 raft_consensus.cc:383] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.641927 1154 raft_consensus.cc:738] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.642139 1154 consensus_queue.cc:260] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.642670 1154 ts_tablet_manager.cc:1428] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:04.642925 1154 tablet_bootstrap.cc:492] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.643136 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01d95356c7a7419bb56ddf67d451eadc" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:04.643314 1048 raft_consensus.cc:3058] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.644212 1154 tablet_bootstrap.cc:654] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.646818 710 leader_election.cc:304] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.647073 2548 raft_consensus.cc:2802] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.647204 2548 raft_consensus.cc:695] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:04.647529 2548 consensus_queue.cc:237] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.657272 1154 tablet_bootstrap.cc:492] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.657424 1154 ts_tablet_manager.cc:1397] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20250629 01:52:04.658017 1154 raft_consensus.cc:357] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.658185 1154 raft_consensus.cc:383] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.658257 1154 raft_consensus.cc:738] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.658456 1154 consensus_queue.cc:260] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.659458 1154 ts_tablet_manager.cc:1428] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250629 01:52:04.659825 1154 tablet_bootstrap.cc:492] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.661375 1154 tablet_bootstrap.cc:654] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:04.663609 1155 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.225s sys 0.260s
W20250629 01:52:04.663739 1155 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.713s user 0.225s sys 0.260s
I20250629 01:52:04.669334 1155 tablet_bootstrap.cc:492] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:04.669440 1155 ts_tablet_manager.cc:1397] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.721s user 0.226s sys 0.260s
I20250629 01:52:04.669967 1155 raft_consensus.cc:357] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.670090 1155 raft_consensus.cc:383] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.670127 1155 raft_consensus.cc:738] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.670256 1155 consensus_queue.cc:260] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.670645 1155 ts_tablet_manager.cc:1428] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:04.670872 1155 tablet_bootstrap.cc:492] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:04.672027 1155 tablet_bootstrap.cc:654] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.678838 2980 tablet.cc:1722] T 3da0deb865984f70913323364ca38703 P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250629 01:52:04.683809 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet dc841eb171d04b118b1e705ceed4c861 (table default.Table_7f95c64f736b4aee86abc0c7785ab8d6 [id=0429a74c16854b3092e37a247f33c497]) successfully deleted
I20250629 01:52:04.685900 2966 raft_consensus.cc:491] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.686062 2966 raft_consensus.cc:513] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.686241 916 raft_consensus.cc:3058] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.686710 2966 leader_election.cc:290] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.687393 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cced2e8c94345b88f388becb0060d7a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.687564 1044 raft_consensus.cc:2466] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:04.688189 710 leader_election.cc:304] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:04.688432 2966 raft_consensus.cc:2802] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.688436 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cced2e8c94345b88f388becb0060d7a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:04.688530 2966 raft_consensus.cc:491] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.688614 2966 raft_consensus.cc:3058] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:04.688913 712 leader_election.cc:343] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.693426 1155 tablet_bootstrap.cc:492] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:04.693625 1155 ts_tablet_manager.cc:1397] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.002s
I20250629 01:52:04.694586 1155 raft_consensus.cc:357] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.694772 1155 raft_consensus.cc:383] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.694855 1155 raft_consensus.cc:738] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.695034 1155 consensus_queue.cc:260] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.695708 1155 ts_tablet_manager.cc:1428] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:04.695966 1155 tablet_bootstrap.cc:492] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:04.697686 1155 tablet_bootstrap.cc:654] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.699172 1154 tablet_bootstrap.cc:492] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.699348 1154 ts_tablet_manager.cc:1397] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.010s
I20250629 01:52:04.700143 1154 raft_consensus.cc:357] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.700325 1154 raft_consensus.cc:383] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.700426 1154 raft_consensus.cc:738] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.700750 1154 consensus_queue.cc:260] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.701753 1154 ts_tablet_manager.cc:1428] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:04.702438 1154 tablet_bootstrap.cc:492] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.704058 1154 tablet_bootstrap.cc:654] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.705139 907 raft_consensus.cc:3058] T d062fb830020442faee626be006926c1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.708943 1159 tablet_bootstrap.cc:492] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.709072 1159 ts_tablet_manager.cc:1397] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.405s user 0.012s sys 0.013s
I20250629 01:52:04.709726 1159 raft_consensus.cc:357] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.709862 1159 raft_consensus.cc:383] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.709911 1159 raft_consensus.cc:738] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.710057 1159 consensus_queue.cc:260] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.710553 1159 ts_tablet_manager.cc:1428] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:04.711259 1159 tablet_bootstrap.cc:492] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.712514 1159 tablet_bootstrap.cc:654] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.712934 1154 tablet_bootstrap.cc:492] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.713083 1154 ts_tablet_manager.cc:1397] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20250629 01:52:04.713948 1154 raft_consensus.cc:357] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.714370 1154 raft_consensus.cc:383] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.714514 1154 raft_consensus.cc:738] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.714916 1154 consensus_queue.cc:260] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.716257 1154 ts_tablet_manager.cc:1428] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:52:04.717386 1154 tablet_bootstrap.cc:492] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.719165 1154 tablet_bootstrap.cc:654] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.721096 1155 tablet_bootstrap.cc:492] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:04.721249 1155 ts_tablet_manager.cc:1397] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.007s
I20250629 01:52:04.721848 1155 raft_consensus.cc:357] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.722030 1155 raft_consensus.cc:383] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.722107 1155 raft_consensus.cc:738] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.722282 1155 consensus_queue.cc:260] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.722708 1155 ts_tablet_manager.cc:1428] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250629 01:52:04.723064 1155 tablet_bootstrap.cc:492] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:04.724730 1155 tablet_bootstrap.cc:654] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.722630 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1462) took 1284 ms. Trace:
I20250629 01:52:04.725288 1158 rpcz_store.cc:276] 0629 01:52:03.438178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.438215 (+ 37us) service_pool.cc:225] Handling call
0629 01:52:04.352394 (+914179us) ts_tablet_manager.cc:1145] Deleting tablet 67e90a48af8f48bba3b3fd4f4110d367
0629 01:52:04.352400 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.482676 (+130276us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.636115 (+153439us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.722616 (+ 86501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":914056}
I20250629 01:52:04.725907 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 67e90a48af8f48bba3b3fd4f4110d367 (table default.Table_eacbe3d626eb4bb980d04c2c6647aec9 [id=22db5880e03948678f7bcf4aa8082579]) successfully deleted
I20250629 01:52:04.726280 1158 tablet_replica.cc:331] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:04.726505 1158 raft_consensus.cc:2241] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:04.726698 1158 raft_consensus.cc:2270] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.727216 1158 ts_tablet_manager.cc:1905] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.731384 1156 ts_tablet_manager.cc:1918] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:04.731470 1156 log.cc:1199] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/4aa8dfd712c043ad9f8ff56b064bb8fd
I20250629 01:52:04.731791 1156 ts_tablet_manager.cc:1939] T 4aa8dfd712c043ad9f8ff56b064bb8fd P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:04.733116 1048 raft_consensus.cc:2466] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:04.740890 2966 raft_consensus.cc:513] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.741078 2990 tablet.cc:1722] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250629 01:52:04.744837 2966 leader_election.cc:290] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.745195 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cced2e8c94345b88f388becb0060d7a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:04.745585 712 leader_election.cc:343] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.755725 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8cced2e8c94345b88f388becb0060d7a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:04.755899 1048 raft_consensus.cc:3058] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.756543 1159 tablet_bootstrap.cc:492] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.756697 1159 ts_tablet_manager.cc:1397] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.001s
I20250629 01:52:04.757272 1159 raft_consensus.cc:357] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.757475 1159 raft_consensus.cc:383] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.757555 1159 raft_consensus.cc:738] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
W20250629 01:52:04.760021 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c8ad1108c6f94621a65cd5d9e3c512fc consensus state has no leader
I20250629 01:52:04.760286 1159 consensus_queue.cc:260] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.761266 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d6ced5759a50401a9d6c4bf075d94732
I20250629 01:52:04.761721 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bba4af011c78409aaf9dfc00191c2e5b
I20250629 01:52:04.761994 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_7fccef05c28f4b9d84145aeeca6acb0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.762488 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fccef05c28f4b9d84145aeeca6acb0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.762826 451 catalog_manager.cc:5582] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4712643b602e4f898752fcab6a1e5913 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4712643b602e4f898752fcab6a1e5913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: HEALTHY } } }
I20250629 01:52:04.763841 1159 ts_tablet_manager.cc:1428] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250629 01:52:04.764070 1159 tablet_bootstrap.cc:492] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.765276 1159 tablet_bootstrap.cc:654] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.766086 1154 tablet_bootstrap.cc:492] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.766180 1154 ts_tablet_manager.cc:1397] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.012s
I20250629 01:52:04.766736 1154 raft_consensus.cc:357] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.766851 1154 raft_consensus.cc:383] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.766907 1154 raft_consensus.cc:738] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.767063 1154 consensus_queue.cc:260] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.769418 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2ae451bc02904676bd61b6aae99eb312"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.769825 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ae451bc02904676bd61b6aae99eb312 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.771157 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ced5759a50401a9d6c4bf075d94732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.771325 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 77727efeef5e4f0c89f4afca2941d02a
I20250629 01:52:04.771829 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_674ede5eaf0d47ae806ff88f77980783"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:04.772223 1155 tablet_bootstrap.cc:492] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
W20250629 01:52:04.772235 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_674ede5eaf0d47ae806ff88f77980783 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.772332 1155 ts_tablet_manager.cc:1397] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.008s
I20250629 01:52:04.772935 1155 raft_consensus.cc:357] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.773074 1155 raft_consensus.cc:383] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.773128 1155 raft_consensus.cc:738] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.773633 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 77727efeef5e4f0c89f4afca2941d02a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
W20250629 01:52:04.773763 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2213f74c54534046a671808db21f6ee8 consensus state has no leader
I20250629 01:52:04.774106 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_ab0ea69604d14932ac24e524c6ad7c5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.774544 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab0ea69604d14932ac24e524c6ad7c5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.775699 1154 ts_tablet_manager.cc:1428] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250629 01:52:04.776320 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2fa469b443b4b3e8f3e2d272e328a4a" } modify_external_catalogs: true
I20250629 01:52:04.776466 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f2fa469b443b4b3e8f3e2d272e328a4a" } modify_external_catalogs: true
I20250629 01:52:04.776684 1155 consensus_queue.cc:260] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.776204 1154 tablet_bootstrap.cc:492] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.778525 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1966a23b1fe14e6a94bd24ee4e8447a0
I20250629 01:52:04.779471 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a85a0ce70b4e4a8a95d40d3423e0746e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.779938 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a85a0ce70b4e4a8a95d40d3423e0746e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.779994 1159 tablet_bootstrap.cc:492] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.780086 1159 ts_tablet_manager.cc:1397] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250629 01:52:04.777197 1155 ts_tablet_manager.cc:1428] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250629 01:52:04.780628 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 1966a23b1fe14e6a94bd24ee4e8447a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:04.781025 1159 raft_consensus.cc:357] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.781211 1159 raft_consensus.cc:383] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.781306 1159 raft_consensus.cc:738] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.781531 1159 consensus_queue.cc:260] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.782095 1159 ts_tablet_manager.cc:1428] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:04.782352 1159 tablet_bootstrap.cc:492] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.783509 1159 tablet_bootstrap.cc:654] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.783573 1154 tablet_bootstrap.cc:654] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.795003 1159 tablet_bootstrap.cc:492] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.795148 1159 ts_tablet_manager.cc:1397] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250629 01:52:04.795765 1159 raft_consensus.cc:357] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.795953 1159 raft_consensus.cc:383] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.796033 1159 raft_consensus.cc:738] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.796283 1159 consensus_queue.cc:260] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:04.796801 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec0365d5ad654a7a9df622a27bfbe6fd consensus state has no leader
I20250629 01:52:04.796913 1159 ts_tablet_manager.cc:1428] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:04.797109 1159 tablet_bootstrap.cc:492] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.798311 1159 tablet_bootstrap.cc:654] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.799456 710 leader_election.cc:304] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.801424 2971 raft_consensus.cc:2802] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.801662 2971 raft_consensus.cc:695] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:04.802157 2971 consensus_queue.cc:237] T 01d95356c7a7419bb56ddf67d451eadc P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.802595 1154 tablet_bootstrap.cc:492] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.802737 1154 ts_tablet_manager.cc:1397] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250629 01:52:04.803369 1154 raft_consensus.cc:357] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.803606 1154 raft_consensus.cc:383] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.803676 1154 raft_consensus.cc:738] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.803985 1154 consensus_queue.cc:260] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.804678 1154 ts_tablet_manager.cc:1428] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250629 01:52:04.808540 1154 tablet_bootstrap.cc:492] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.810300 1154 tablet_bootstrap.cc:654] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.818163 1154 tablet_bootstrap.cc:492] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.818297 1154 ts_tablet_manager.cc:1397] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20250629 01:52:04.818885 2966 raft_consensus.cc:491] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.818878 1154 raft_consensus.cc:357] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.819026 1154 raft_consensus.cc:383] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.819083 1154 raft_consensus.cc:738] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.819032 2966 raft_consensus.cc:513] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.819239 1154 consensus_queue.cc:260] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.819521 2966 leader_election.cc:290] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:04.819926 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f465d0bb46674763abfc8d93f45cd13d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.820484 712 leader_election.cc:343] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.821955 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b8599c95728d43dd898fbdfcb4fe5db3" } new_table_name: "default.Table_520e60b731224e1b98a5165831c7148f" modify_external_catalogs: true
I20250629 01:52:04.822137 1154 ts_tablet_manager.cc:1428] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250629 01:52:04.822470 1154 tablet_bootstrap.cc:492] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.823717 1154 tablet_bootstrap.cc:654] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.828477 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 77727efeef5e4f0c89f4afca2941d02a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.829908 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 1966a23b1fe14e6a94bd24ee4e8447a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.830395 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f465d0bb46674763abfc8d93f45cd13d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.830535 1044 raft_consensus.cc:2466] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:04.830641 745 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ced5759a50401a9d6c4bf075d94732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:04.831588 1155 tablet_bootstrap.cc:492] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:04.831881 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1437) took 2694 ms. Trace:
I20250629 01:52:04.831959 1156 rpcz_store.cc:276] 0629 01:52:02.137689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.137733 (+ 44us) service_pool.cc:225] Handling call
0629 01:52:04.543933 (+2406200us) ts_tablet_manager.cc:1145] Deleting tablet 4aa8dfd712c043ad9f8ff56b064bb8fd
0629 01:52:04.543941 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.624986 (+ 81045us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.731288 (+106302us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.831838 (+100550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2406060}
I20250629 01:52:04.832274 1156 tablet_replica.cc:331] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:04.832489 1156 raft_consensus.cc:2241] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:04.832743 1156 raft_consensus.cc:2270] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:04.832981 1155 tablet_bootstrap.cc:654] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.833523 1159 tablet_bootstrap.cc:492] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.833650 1159 ts_tablet_manager.cc:1397] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20250629 01:52:04.833972 1156 ts_tablet_manager.cc:1905] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:04.834411 1159 raft_consensus.cc:357] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.834574 1159 raft_consensus.cc:383] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.834635 1159 raft_consensus.cc:738] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.834820 1159 consensus_queue.cc:260] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.835333 1159 ts_tablet_manager.cc:1428] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250629 01:52:04.835549 1159 tablet_bootstrap.cc:492] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.836943 1159 tablet_bootstrap.cc:654] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.838011 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 4aa8dfd712c043ad9f8ff56b064bb8fd (table default.Table_df8a368b4df84492add9218c8d1deb22 [id=c03cb89aed104caa852122ebd5722929]) successfully deleted
I20250629 01:52:04.840411 710 leader_election.cc:304] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.841149 2548 raft_consensus.cc:2802] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.841320 2548 raft_consensus.cc:491] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.841507 2548 raft_consensus.cc:3058] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.842530 1159 tablet_bootstrap.cc:492] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.842631 1159 ts_tablet_manager.cc:1397] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20250629 01:52:04.843165 1159 raft_consensus.cc:357] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.843300 1159 raft_consensus.cc:383] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.843349 1159 raft_consensus.cc:738] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.843537 1159 consensus_queue.cc:260] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.844568 1159 ts_tablet_manager.cc:1428] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250629 01:52:04.845556 1159 tablet_bootstrap.cc:492] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.846889 1159 tablet_bootstrap.cc:654] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.847988 2896 raft_consensus.cc:491] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.848111 2896 raft_consensus.cc:513] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.848532 2896 leader_election.cc:290] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:04.849021 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:04.849082 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4030891de41444e0b387492e570a11a5" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.849541 850 leader_election.cc:304] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.849840 2896 raft_consensus.cc:2747] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.853206 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b9cb2b777214494e9a98a81d9b7785ee" } new_table_name: "default.Table_dbf231f68cde4f7389c64d2bac6734f3" modify_external_catalogs: true
I20250629 01:52:04.856390 1159 tablet_bootstrap.cc:492] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.856503 1159 ts_tablet_manager.cc:1397] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20250629 01:52:04.857151 1159 raft_consensus.cc:357] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.857304 1159 raft_consensus.cc:383] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.857364 1159 raft_consensus.cc:738] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.859076 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_53a80c83ca7d41d28a91bfcc4d622e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.859544 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53a80c83ca7d41d28a91bfcc4d622e89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.860801 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 0f05d54fc9e649faabee85a418ebf2bb from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:04.861326 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_70de6445cd594ad1ae70836bb1749bfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:04.861692 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70de6445cd594ad1ae70836bb1749bfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:04.864276 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 999a62f27eb441699f641562ae60a287 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:04.868935 2859 ts_tablet_manager.cc:1918] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.869065 2859 log.cc:1199] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/5b8a2d1a7f7146f4968590914374ba23
I20250629 01:52:04.869365 2859 ts_tablet_manager.cc:1939] T 5b8a2d1a7f7146f4968590914374ba23 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:04.870162 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d6ced5759a50401a9d6c4bf075d94732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.870326 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 1966a23b1fe14e6a94bd24ee4e8447a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.871102 2758 raft_consensus.cc:491] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.871204 2758 raft_consensus.cc:513] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.871619 2758 leader_election.cc:290] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.872164 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5e9646c3ad04ed580f1d3402010dd38" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:04.872336 772 raft_consensus.cc:2466] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:04.872710 986 leader_election.cc:304] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:04.872982 2758 raft_consensus.cc:2802] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.873042 2758 raft_consensus.cc:491] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.873088 2758 raft_consensus.cc:3058] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.880044 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5e9646c3ad04ed580f1d3402010dd38" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:04.880245 1159 consensus_queue.cc:260] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:04.880650 988 leader_election.cc:343] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.881007 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 77727efeef5e4f0c89f4afca2941d02a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:04.882692 1159 ts_tablet_manager.cc:1428] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250629 01:52:04.883754 1159 tablet_bootstrap.cc:492] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.884253 1048 raft_consensus.cc:2466] T 8cced2e8c94345b88f388becb0060d7a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:04.890445 1159 tablet_bootstrap.cc:654] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.902323 1159 tablet_bootstrap.cc:492] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.902424 1159 ts_tablet_manager.cc:1397] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250629 01:52:04.902980 1159 raft_consensus.cc:357] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.903113 1159 raft_consensus.cc:383] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.903158 1159 raft_consensus.cc:738] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.904076 2651 raft_consensus.cc:491] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.904306 2651 raft_consensus.cc:513] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.905104 2651 leader_election.cc:290] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.905400 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "056eeff228974a8e9036a37b58a2a675" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:04.905792 712 leader_election.cc:343] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.906327 1159 consensus_queue.cc:260] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.909500 1159 ts_tablet_manager.cc:1428] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250629 01:52:04.914745 1159 tablet_bootstrap.cc:492] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.916258 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "056eeff228974a8e9036a37b58a2a675" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:04.916491 1048 raft_consensus.cc:2466] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:04.918226 1159 tablet_bootstrap.cc:654] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.918705 710 leader_election.cc:304] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.919800 2963 raft_consensus.cc:2802] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:04.920423 2963 raft_consensus.cc:695] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:04.920670 2963 consensus_queue.cc:237] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.921818 710 leader_election.cc:304] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:04.922106 2534 raft_consensus.cc:2802] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:04.922184 2534 raft_consensus.cc:491] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:04.922267 2534 raft_consensus.cc:3058] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.930759 1155 tablet_bootstrap.cc:492] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:04.930945 1155 ts_tablet_manager.cc:1397] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.015s
I20250629 01:52:04.931403 2896 raft_consensus.cc:491] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.931661 1155 raft_consensus.cc:357] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.931525 2896 raft_consensus.cc:513] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.931813 1155 raft_consensus.cc:383] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.931883 1155 raft_consensus.cc:738] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.932061 1155 consensus_queue.cc:260] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.932351 2758 raft_consensus.cc:513] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.932533 1155 ts_tablet_manager.cc:1428] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:04.932763 1155 tablet_bootstrap.cc:492] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:04.932876 2758 leader_election.cc:290] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.933257 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5e9646c3ad04ed580f1d3402010dd38" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:04.933459 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5e9646c3ad04ed580f1d3402010dd38" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:04.933594 772 raft_consensus.cc:3058] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.933853 1158 ts_tablet_manager.cc:1918] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:04.933938 1158 log.cc:1199] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/ef3060d46ac24330be49103920dc1dee
I20250629 01:52:04.934203 1158 ts_tablet_manager.cc:1939] T ef3060d46ac24330be49103920dc1dee P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:52:04.934274 988 leader_election.cc:343] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.934649 2990 tablet.cc:1717] T 76118a6e684448b09f244d44c909ccaa P 443c4bd3c5d548ee8cd347d2b4de5293: Skipping requested alter to schema version 5, tablet already version 5
I20250629 01:52:04.935923 1156 ts_tablet_manager.cc:1918] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:04.936043 1156 log.cc:1199] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/c5d45854191247bf8172d0bbfd01b6db
I20250629 01:52:04.936411 1156 ts_tablet_manager.cc:1939] T c5d45854191247bf8172d0bbfd01b6db P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:04.937556 2896 leader_election.cc:290] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:04.938925 850 leader_election.cc:343] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 07e0727102d54294a7c13a4aeef311e1
W20250629 01:52:04.940143 850 leader_election.cc:343] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 07e0727102d54294a7c13a4aeef311e1
I20250629 01:52:04.940248 850 leader_election.cc:304] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.940681 2896 raft_consensus.cc:2747] T 07e0727102d54294a7c13a4aeef311e1 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.943549 1154 tablet_bootstrap.cc:492] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.943706 1154 ts_tablet_manager.cc:1397] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.013s
I20250629 01:52:04.944413 1155 tablet_bootstrap.cc:654] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.944417 1154 raft_consensus.cc:357] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.944651 1154 raft_consensus.cc:383] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.945175 1154 raft_consensus.cc:738] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.945410 1154 consensus_queue.cc:260] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.948681 1154 ts_tablet_manager.cc:1428] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250629 01:52:04.948701 2548 raft_consensus.cc:513] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.949229 2548 leader_election.cc:290] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:04.949502 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f465d0bb46674763abfc8d93f45cd13d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:04.949990 712 leader_election.cc:343] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:04.950814 1154 tablet_bootstrap.cc:492] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:04.952271 1154 tablet_bootstrap.cc:654] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.959237 1155 tablet_bootstrap.cc:492] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:04.959369 1155 ts_tablet_manager.cc:1397] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.002s
I20250629 01:52:04.960451 1155 raft_consensus.cc:357] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.960639 1155 raft_consensus.cc:383] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.960726 1155 raft_consensus.cc:738] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.960938 1155 consensus_queue.cc:260] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.962111 1155 ts_tablet_manager.cc:1428] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:52:04.962384 1155 tablet_bootstrap.cc:492] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:04.965533 1159 tablet_bootstrap.cc:492] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:04.965687 1159 ts_tablet_manager.cc:1397] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.001s
I20250629 01:52:04.957700 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f465d0bb46674763abfc8d93f45cd13d" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:04.965895 1052 raft_consensus.cc:3058] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:04.967054 1155 tablet_bootstrap.cc:654] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.968521 1159 raft_consensus.cc:357] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:04.970186 712 consensus_peers.cc:489] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:04.968701 1159 raft_consensus.cc:383] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.970420 1159 raft_consensus.cc:738] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.970651 1159 consensus_queue.cc:260] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.972036 2896 raft_consensus.cc:491] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:04.972150 2896 raft_consensus.cc:513] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:04.972535 2896 leader_election.cc:290] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:04.972512 442 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:35873, user_credentials={real_user=slave}} blocked reactor thread for 102686us
W20250629 01:52:04.973268 850 leader_election.cc:343] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 4aa8dfd712c043ad9f8ff56b064bb8fd
W20250629 01:52:04.973863 850 leader_election.cc:343] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 4aa8dfd712c043ad9f8ff56b064bb8fd
I20250629 01:52:04.973956 850 leader_election.cc:304] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:04.974268 2896 raft_consensus.cc:2747] T 4aa8dfd712c043ad9f8ff56b064bb8fd P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:04.974457 1159 ts_tablet_manager.cc:1428] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250629 01:52:04.974988 1159 tablet_bootstrap.cc:492] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:04.980389 1159 tablet_bootstrap.cc:654] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:04.983143 1155 tablet_bootstrap.cc:492] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:04.985786 1155 ts_tablet_manager.cc:1397] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250629 01:52:04.986482 1155 raft_consensus.cc:357] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.986630 1155 raft_consensus.cc:383] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.986699 1155 raft_consensus.cc:738] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.986912 1155 consensus_queue.cc:260] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.988519 1155 ts_tablet_manager.cc:1428] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250629 01:52:04.996224 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: default.Table_5683602f0dce4ff39355c89a0cb64494 [id=80262dd3324a4c2390da9adb2540dd62] alter complete (version 2)
I20250629 01:52:04.997009 1154 tablet_bootstrap.cc:492] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:04.997108 454 master_service.cc:948] ReplaceTablet: received request to replace tablet d331a13d13f34456be057dd9cc9fc23b from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:04.997216 451 master_service.cc:948] ReplaceTablet: received request to replace tablet d4b4e75281ba4cd8a8e1f0a47710964f from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:04.997131 1154 ts_tablet_manager.cc:1397] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250629 01:52:04.998375 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_42b1bfd299f241f69f9ab0fc3ba89196" } new_table_name: "default.Table_96609622f64847039f976cc3e5e6d406" modify_external_catalogs: true
I20250629 01:52:04.998028 1154 raft_consensus.cc:357] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.998772 1154 raft_consensus.cc:383] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:04.998823 1154 raft_consensus.cc:738] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:04.998972 1154 consensus_queue.cc:260] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:04.999361 1154 ts_tablet_manager.cc:1428] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:52:04.999418 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 34a72207d9134115b98a334424974cc8
I20250629 01:52:04.999966 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20b2067750ab47deae63c0f954ef7683" } modify_external_catalogs: true
I20250629 01:52:05.000283 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a72207d9134115b98a334424974cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.000310 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_13401ad7d0e2429a8e31aff8e7a61a3e" } modify_external_catalogs: true
I20250629 01:52:05.000407 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_13401ad7d0e2429a8e31aff8e7a61a3e" } modify_external_catalogs: true
I20250629 01:52:05.000453 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_20b2067750ab47deae63c0f954ef7683" } modify_external_catalogs: true
I20250629 01:52:05.000829 1154 tablet_bootstrap.cc:492] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:52:05.001436 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5c8e3cdfbb743438fd806f1e2dd5bf7 consensus state has no leader
I20250629 01:52:05.001688 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_19d55c4c3f444c41b10920604792dccb" } new_table_name: "default.Table_b447771a3e384a5c91b039149027869e" modify_external_catalogs: true
I20250629 01:52:05.002173 1154 tablet_bootstrap.cc:654] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:05.002692 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6464fd2fa5bf46dabf3a177d5e756b94 consensus state has no leader
I20250629 01:52:05.002951 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_65942b65f83f4b2598721e0d8c1ea566"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.003450 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65942b65f83f4b2598721e0d8c1ea566 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.004122 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 999a62f27eb441699f641562ae60a287
I20250629 01:52:05.004521 450 catalog_manager.cc:6652] ReplaceTablet: tablet 999a62f27eb441699f641562ae60a287 deleted and replaced by tablet d72cac859c3b43589e033f605b7e35ad
I20250629 01:52:05.004734 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71daf3fef0d74acca9e43c6182e18de7" } modify_external_catalogs: true
I20250629 01:52:05.004848 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_71daf3fef0d74acca9e43c6182e18de7" } modify_external_catalogs: true
I20250629 01:52:05.005023 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 999a62f27eb441699f641562ae60a287 with delete_type TABLET_DATA_DELETED (replaced by tablet d72cac859c3b43589e033f605b7e35ad) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.005388 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 557ecee0d6e84f77ad10bbdd27859ee3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.005561 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a72207d9134115b98a334424974cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.005775 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 999a62f27eb441699f641562ae60a287 with delete_type TABLET_DATA_DELETED (replaced by tablet d72cac859c3b43589e033f605b7e35ad) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.005993 2758 raft_consensus.cc:491] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.006100 2758 raft_consensus.cc:513] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.006598 2758 leader_election.cc:290] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:04.999984 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_09b1496250654f8583e789cec2305453" } modify_external_catalogs: true
I20250629 01:52:05.007355 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_09b1496250654f8583e789cec2305453" } modify_external_catalogs: true
I20250629 01:52:05.007680 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2591c65d53449c8801778606f1f05f1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.008066 988 leader_election.cc:343] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.009999 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2591c65d53449c8801778606f1f05f1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:05.010462 986 leader_election.cc:343] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.010550 986 leader_election.cc:304] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.010895 2758 raft_consensus.cc:2747] T c2591c65d53449c8801778606f1f05f1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.017596 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1463) took 1579 ms. Trace:
I20250629 01:52:05.023155 1052 raft_consensus.cc:2466] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:05.024763 879 tablet_service.cc:1515] Processing DeleteTablet for tablet e5d27d658892496a9b19b57949a8399a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:54 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.024932 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 999a62f27eb441699f641562ae60a287 with delete_type TABLET_DATA_DELETED (replaced by tablet d72cac859c3b43589e033f605b7e35ad) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.025146 2896 raft_consensus.cc:491] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.025279 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 34a72207d9134115b98a334424974cc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.025264 2896 raft_consensus.cc:513] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:05.025647 710 consensus_peers.cc:489] T ef3060d46ac24330be49103920dc1dee P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Couldn't send request to peer 4712643b602e4f898752fcab6a1e5913. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:05.025733 2896 leader_election.cc:290] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.025835 710 leader_election.cc:304] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.026329 2971 raft_consensus.cc:2802] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:05.026288 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a582d4f907f340179024c0377d3893e2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:05.026796 2971 raft_consensus.cc:695] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:05.027284 2971 consensus_queue.cc:237] T f465d0bb46674763abfc8d93f45cd13d P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:05.036203 1155 tablet_bootstrap.cc:492] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:05.037978 1155 tablet_bootstrap.cc:654] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.043602 772 raft_consensus.cc:2466] T c5e9646c3ad04ed580f1d3402010dd38 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:05.043955 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a582d4f907f340179024c0377d3893e2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.044129 986 leader_election.cc:304] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.044559 850 leader_election.cc:304] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.017688 1158 rpcz_store.cc:276] 0629 01:52:03.438537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.438570 (+ 33us) service_pool.cc:225] Handling call
0629 01:52:04.726234 (+1287664us) ts_tablet_manager.cc:1145] Deleting tablet ef3060d46ac24330be49103920dc1dee
0629 01:52:04.726245 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.874670 (+148425us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.933765 (+ 59095us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.017583 (+ 83818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1287551}
I20250629 01:52:05.044867 2896 raft_consensus.cc:2747] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.044924 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet ef3060d46ac24330be49103920dc1dee (table default.Table_f5ba36b31e8a415b81ec003950152c56 [id=f620d7fa33e54d13ada1be914ed57767]) successfully deleted
I20250629 01:52:05.045156 2898 raft_consensus.cc:2802] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:05.045243 2898 raft_consensus.cc:695] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:05.045421 2898 consensus_queue.cc:237] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:05.045733 442 consensus_peers.cc:489] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae -> Peer 0f4258c004e04bcf8216bb7c8fd56c54 (127.0.104.61:35873): Couldn't send request to peer 0f4258c004e04bcf8216bb7c8fd56c54. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:35873: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250629 01:52:05.046206 1158 tablet_replica.cc:331] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:05.046800 1158 raft_consensus.cc:2241] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:05.047067 1158 raft_consensus.cc:2270] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.048116 1158 ts_tablet_manager.cc:1905] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.052068 1155 tablet_bootstrap.cc:492] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:05.052206 1155 ts_tablet_manager.cc:1397] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.001s
I20250629 01:52:05.052774 1155 raft_consensus.cc:357] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.052925 1155 raft_consensus.cc:383] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.052984 1155 raft_consensus.cc:738] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.053205 1155 consensus_queue.cc:260] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.053589 1155 ts_tablet_manager.cc:1428] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250629 01:52:05.053813 1155 tablet_bootstrap.cc:492] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:05.055444 1155 tablet_bootstrap.cc:654] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.057934 1158 ts_tablet_manager.cc:1918] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:05.057945 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1396) took 1920 ms. Trace:
I20250629 01:52:05.058074 1158 log.cc:1199] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/12b85f220f3947ca8110f2db937e7b68
I20250629 01:52:05.058010 2859 rpcz_store.cc:276] 0629 01:52:03.137882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.137928 (+ 46us) service_pool.cc:225] Handling call
0629 01:52:04.545351 (+1407423us) ts_tablet_manager.cc:1145] Deleting tablet 5b8a2d1a7f7146f4968590914374ba23
0629 01:52:04.545358 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.666043 (+120685us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.868831 (+202788us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.057929 (+189098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1407290}
I20250629 01:52:05.058355 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 5b8a2d1a7f7146f4968590914374ba23 (table default.Table_98a1aa99dbcb4f448201bde47757f522 [id=5ee5e560e23a45ff897dbba0a0909cda]) successfully deleted
I20250629 01:52:05.058575 1158 ts_tablet_manager.cc:1939] T 12b85f220f3947ca8110f2db937e7b68 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:05.059209 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1464) took 1620 ms. Trace:
I20250629 01:52:05.059324 1158 rpcz_store.cc:276] 0629 01:52:03.438902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.438936 (+ 34us) service_pool.cc:225] Handling call
0629 01:52:05.044788 (+1605852us) ts_tablet_manager.cc:1145] Deleting tablet 12b85f220f3947ca8110f2db937e7b68
0629 01:52:05.046164 (+ 1376us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.051123 (+ 4959us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.057872 (+ 6749us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.059195 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1605744}
I20250629 01:52:05.059737 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 12b85f220f3947ca8110f2db937e7b68 (table default.Table_13f7043c741546969926f8ab0a5b0782 [id=194bdf24776b480895832c6d00becb73]) successfully deleted
I20250629 01:52:05.060016 1158 tablet_replica.cc:331] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:05.060231 1158 raft_consensus.cc:2241] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.060398 1158 raft_consensus.cc:2270] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.061116 1158 ts_tablet_manager.cc:1905] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.067438 2894 consensus_queue.cc:1035] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:05.076689 2896 raft_consensus.cc:491] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.076789 2896 raft_consensus.cc:513] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.077059 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1397) took 1936 ms. Trace:
I20250629 01:52:05.077122 2859 rpcz_store.cc:276] 0629 01:52:03.140716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.140940 (+ 224us) service_pool.cc:225] Handling call
0629 01:52:05.076949 (+1936009us) ts_tablet_manager.cc:1145] Deleting tablet 4c07a1580c734aeba695151d23c7e020
0629 01:52:05.076959 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.077045 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1917223}
I20250629 01:52:05.077196 2896 leader_election.cc:290] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.077322 2859 tablet_replica.cc:331] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
W20250629 01:52:05.077519 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 4c07a1580c734aeba695151d23c7e020 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c07a1580c734aeba695151d23c7e020 already in progress: creating tablet
I20250629 01:52:05.078200 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1438) took 2940 ms. Trace:
I20250629 01:52:05.078274 1156 rpcz_store.cc:276] 0629 01:52:02.138072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.138121 (+ 49us) service_pool.cc:225] Handling call
0629 01:52:04.832209 (+2694088us) ts_tablet_manager.cc:1145] Deleting tablet c5d45854191247bf8172d0bbfd01b6db
0629 01:52:04.832219 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:04.895886 (+ 63667us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:04.935755 (+ 39869us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.078167 (+142412us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2693928}
I20250629 01:52:05.078537 1156 tablet_replica.cc:331] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:05.078684 1156 raft_consensus.cc:2241] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:05.078917 1156 pending_rounds.cc:70] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Trying to abort 1 pending ops.
I20250629 01:52:05.078964 1156 pending_rounds.cc:77] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7172759232763793408 op_type: NO_OP noop_request { }
I20250629 01:52:05.079036 1156 raft_consensus.cc:2887] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250629 01:52:05.079104 1156 raft_consensus.cc:2270] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.079511 1156 ts_tablet_manager.cc:1905] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.080263 2859 raft_consensus.cc:2241] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.080406 2859 raft_consensus.cc:2270] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.080478 2534 raft_consensus.cc:513] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.080828 2859 ts_tablet_manager.cc:1905] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.080987 2534 leader_election.cc:290] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.081171 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet c5d45854191247bf8172d0bbfd01b6db (table default.Table_4c419cdd9add4db7b041836405c4856b [id=7f7fa7fcbfc248319d2a0b8085312cdb]) successfully deleted
I20250629 01:52:05.081732 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "056eeff228974a8e9036a37b58a2a675" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:05.081856 1052 raft_consensus.cc:3058] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.082176 1155 tablet_bootstrap.cc:492] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:05.082284 1155 ts_tablet_manager.cc:1397] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20250629 01:52:05.082878 1155 raft_consensus.cc:357] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.083070 1155 raft_consensus.cc:383] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.083143 1155 raft_consensus.cc:738] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.083357 1155 consensus_queue.cc:260] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.083899 1155 ts_tablet_manager.cc:1428] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250629 01:52:05.083990 1052 raft_consensus.cc:2466] T 056eeff228974a8e9036a37b58a2a675 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:05.086536 850 leader_election.cc:343] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 50d23f589de84dd5b150a17cee00c411
I20250629 01:52:05.087042 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "056eeff228974a8e9036a37b58a2a675" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:05.088198 850 leader_election.cc:343] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 50d23f589de84dd5b150a17cee00c411
I20250629 01:52:05.088291 850 leader_election.cc:304] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.088615 2896 raft_consensus.cc:2747] T 50d23f589de84dd5b150a17cee00c411 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.089298 907 raft_consensus.cc:3058] T a582d4f907f340179024c0377d3893e2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.092839 916 raft_consensus.cc:3058] T 4030891de41444e0b387492e570a11a5 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.098618 1155 tablet_bootstrap.cc:492] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:05.100561 1158 ts_tablet_manager.cc:1918] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:05.085832 1156 ts_tablet_manager.cc:1918] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250629 01:52:05.088979 712 leader_election.cc:343] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.101323 1156 log.cc:1199] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/3ba5887dffc1460a89208447ecc6a198
I20250629 01:52:05.101347 710 leader_election.cc:304] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.101806 2970 raft_consensus.cc:2802] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:05.101922 2970 raft_consensus.cc:695] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 LEADER]: Becoming Leader. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Running, Role: LEADER
I20250629 01:52:05.102144 2970 consensus_queue.cc:237] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.106968 1156 ts_tablet_manager.cc:1939] T 3ba5887dffc1460a89208447ecc6a198 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:05.112645 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1439) took 2974 ms. Trace:
I20250629 01:52:05.117009 1158 log.cc:1199] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/2ec92eddfabf40918a6cabaff5fe113a
I20250629 01:52:05.117342 1158 ts_tablet_manager.cc:1939] T 2ec92eddfabf40918a6cabaff5fe113a P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:05.106281 1154 tablet_bootstrap.cc:492] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
W20250629 01:52:05.110656 988 consensus_peers.cc:489] T 9ed1ee73bb8f4761806a66f988b57bf9 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:05.117660 1154 ts_tablet_manager.cc:1397] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.117s user 0.003s sys 0.002s
I20250629 01:52:05.117955 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1465) took 1678 ms. Trace:
I20250629 01:52:05.118029 1158 rpcz_store.cc:276] 0629 01:52:03.439262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.439357 (+ 95us) service_pool.cc:225] Handling call
0629 01:52:05.059976 (+1620619us) ts_tablet_manager.cc:1145] Deleting tablet 2ec92eddfabf40918a6cabaff5fe113a
0629 01:52:05.059985 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.063603 (+ 3618us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.100500 (+ 36897us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.117936 (+ 17436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620508}
I20250629 01:52:05.119853 1158 tablet_replica.cc:331] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:05.119949 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 2ec92eddfabf40918a6cabaff5fe113a (table default.Table_96a279d58b0a48b19a894d999572a340 [id=2ca8efc02f174cd8a1a375491a3a48ad]) successfully deleted
I20250629 01:52:05.120028 1158 raft_consensus.cc:2241] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.120159 1158 raft_consensus.cc:2270] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.123687 1158 ts_tablet_manager.cc:1905] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.128546 1156 rpcz_store.cc:276] 0629 01:52:02.138549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.138593 (+ 44us) service_pool.cc:225] Handling call
0629 01:52:05.078491 (+2939898us) ts_tablet_manager.cc:1145] Deleting tablet 3ba5887dffc1460a89208447ecc6a198
0629 01:52:05.078498 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.083371 (+ 4873us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.085751 (+ 2380us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.112615 (+ 26864us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2939767}
I20250629 01:52:05.128903 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 3ba5887dffc1460a89208447ecc6a198 (table default.Table_f3aaa9e9eb034a5cbbd31b1529d0080a [id=7663b74fe2374460b050e5b3fc56ad49]) successfully deleted
I20250629 01:52:05.129166 1156 tablet_replica.cc:331] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:05.129331 1156 raft_consensus.cc:2241] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.129448 1156 raft_consensus.cc:2270] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.129858 1156 ts_tablet_manager.cc:1905] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.134974 1156 ts_tablet_manager.cc:1918] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.135066 1156 log.cc:1199] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/689adaf0c7564d9ba0b31061e4e2d8cf
I20250629 01:52:05.135321 1156 ts_tablet_manager.cc:1939] T 689adaf0c7564d9ba0b31061e4e2d8cf P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:05.118696 1154 raft_consensus.cc:357] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.135581 1154 raft_consensus.cc:383] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.135619 1154 raft_consensus.cc:738] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.135798 1154 consensus_queue.cc:260] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.136191 1154 ts_tablet_manager.cc:1428] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20250629 01:52:05.136402 1154 tablet_bootstrap.cc:492] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
W20250629 01:52:05.137090 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet ec0365d5ad654a7a9df622a27bfbe6fd (table default.Table_761daa0437d34b20900555e0b398a26c [id=0c6d1537cb3d4f0a933d8d2f9a46da65], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ec0365d5ad654a7a9df622a27bfbe6fd consensus state has no leader
I20250629 01:52:05.137964 1154 tablet_bootstrap.cc:654] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.116972 1155 tablet_bootstrap.cc:654] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.145215 1158 ts_tablet_manager.cc:1918] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.145332 1158 log.cc:1199] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/01d95356c7a7419bb56ddf67d451eadc
I20250629 01:52:05.145650 1158 ts_tablet_manager.cc:1939] T 01d95356c7a7419bb56ddf67d451eadc P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:05.146257 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1466) took 1706 ms. Trace:
I20250629 01:52:05.146366 1158 rpcz_store.cc:276] 0629 01:52:03.439684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.439718 (+ 34us) service_pool.cc:225] Handling call
0629 01:52:05.119817 (+1680099us) ts_tablet_manager.cc:1145] Deleting tablet 01d95356c7a7419bb56ddf67d451eadc
0629 01:52:05.119824 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.126367 (+ 6543us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.145153 (+ 18786us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.146243 (+ 1090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679986}
I20250629 01:52:05.146745 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet 01d95356c7a7419bb56ddf67d451eadc (table default.Table_dacb5d3de26e4c82bc85838c8621390e [id=3f6cd9a247f34a6d84c591bca6e7a4f8]) successfully deleted
I20250629 01:52:05.147037 1158 tablet_replica.cc:331] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:05.147222 1158 raft_consensus.cc:2241] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Raft consensus shutting down.
I20250629 01:52:05.147500 1158 raft_consensus.cc:2270] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.148178 1158 ts_tablet_manager.cc:1905] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.154085 1155 tablet_bootstrap.cc:492] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:05.154204 1155 ts_tablet_manager.cc:1397] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.001s
I20250629 01:52:05.154479 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1435) took 3017 ms. Trace:
I20250629 01:52:05.154800 1155 raft_consensus.cc:357] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.156905 1155 raft_consensus.cc:383] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.157013 1155 raft_consensus.cc:738] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.157229 1155 consensus_queue.cc:260] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.158196 1155 ts_tablet_manager.cc:1428] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250629 01:52:05.159125 1155 tablet_bootstrap.cc:492] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:05.159765 2859 ts_tablet_manager.cc:1918] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.159873 2859 log.cc:1199] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/8975c1deaaf44761be791f460b53ec74
I20250629 01:52:05.160187 2859 ts_tablet_manager.cc:1939] T 8975c1deaaf44761be791f460b53ec74 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.160426 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1398) took 2018 ms. Trace:
I20250629 01:52:05.160496 2859 rpcz_store.cc:276] 0629 01:52:03.142120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.142164 (+ 44us) service_pool.cc:225] Handling call
0629 01:52:05.077291 (+1935127us) ts_tablet_manager.cc:1145] Deleting tablet 8975c1deaaf44761be791f460b53ec74
0629 01:52:05.077298 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.100470 (+ 23172us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.159677 (+ 59207us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.160413 (+ 736us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935008}
I20250629 01:52:05.160759 2859 tablet_replica.cc:331] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.160856 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 8975c1deaaf44761be791f460b53ec74 (table default.Table_d532629a716e44368b4324cf8b41f873 [id=21d8cd304488494fb56540f29f71f25d]) successfully deleted
I20250629 01:52:05.160987 2859 raft_consensus.cc:2241] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.161121 2859 raft_consensus.cc:2270] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.161280 1155 tablet_bootstrap.cc:654] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.161563 2859 ts_tablet_manager.cc:1905] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.162009 1156 rpcz_store.cc:276] 0629 01:52:02.137017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.169056 (+ 32039us) service_pool.cc:225] Handling call
0629 01:52:05.129112 (+2960056us) ts_tablet_manager.cc:1145] Deleting tablet 689adaf0c7564d9ba0b31061e4e2d8cf
0629 01:52:05.129120 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.132584 (+ 3464us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.134900 (+ 2316us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.154438 (+ 19538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2959807}
I20250629 01:52:05.162382 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 689adaf0c7564d9ba0b31061e4e2d8cf (table default.Table_55cf597f7eab480299bb4d32ccf5868c [id=fb540507d98341b2801c8394decec635]) successfully deleted
I20250629 01:52:05.162647 1156 tablet_replica.cc:331] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:05.162964 1158 ts_tablet_manager.cc:1918] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:05.163180 1156 raft_consensus.cc:2241] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.163305 1158 log.cc:1199] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/a75d7986152f427d88ed34a443128bc6
I20250629 01:52:05.163466 1156 raft_consensus.cc:2270] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.163789 1158 ts_tablet_manager.cc:1939] T a75d7986152f427d88ed34a443128bc6 P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:05.164333 1156 ts_tablet_manager.cc:1905] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.168324 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1467) took 1728 ms. Trace:
I20250629 01:52:05.171963 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d4b4e75281ba4cd8a8e1f0a47710964f
I20250629 01:52:05.172370 451 catalog_manager.cc:6652] ReplaceTablet: tablet d4b4e75281ba4cd8a8e1f0a47710964f deleted and replaced by tablet 2083db0432024c898f63e5dbff47a2d9
I20250629 01:52:05.172538 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d331a13d13f34456be057dd9cc9fc23b
I20250629 01:52:05.172828 454 catalog_manager.cc:6652] ReplaceTablet: tablet d331a13d13f34456be057dd9cc9fc23b deleted and replaced by tablet 574c51442a6a48ee8af1cbdcdb2b963c
I20250629 01:52:05.175690 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet d062fb830020442faee626be006926c1 (table default.Table_02a7582c640d4b0c966097f23f57cbaf [id=3ab3a1e396af475788b1dd132f662f70]). Expected version 1 got 0
I20250629 01:52:05.175848 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:05.176278 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d4dacbd3dbf34f52bcb81af671ddcf6f" } new_table_name: "default.Table_0f77e24f86614a2e9d585dee76b7e5c4" modify_external_catalogs: true
I20250629 01:52:05.177445 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet feeaa50e7f224da69128ea916e4145fd
I20250629 01:52:05.178030 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e71a853659a04241a66f7c0104379c10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.178527 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e71a853659a04241a66f7c0104379c10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:05.179772 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac0164ca224e40629bfe5ec6006fba1b consensus state has no leader
I20250629 01:52:05.179375 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_3983bb4bae94486bb7be30de3de74aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.180182 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_a4a18be7ca16433aa9ed5bc0d16228ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.180351 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3983bb4bae94486bb7be30de3de74aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:05.180464 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4a18be7ca16433aa9ed5bc0d16228ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:05.181318 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6298871f47a94ff58c9931e0fa913771 consensus state has no leader
I20250629 01:52:05.181545 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_2a4fead6a8a24ca6a78cc5b0367f36f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.181859 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a4fead6a8a24ca6a78cc5b0367f36f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.182687 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d60a756dd2ed4a2e96aaf3f13e3a90de
I20250629 01:52:05.183598 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a2ce14bc48f24499ad1a02fbd53fbe08
I20250629 01:52:05.186041 1159 tablet_bootstrap.cc:492] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.186149 1159 ts_tablet_manager.cc:1397] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.211s user 0.034s sys 0.048s
I20250629 01:52:05.186831 745 tablet_service.cc:1515] Processing DeleteTablet for tablet d4b4e75281ba4cd8a8e1f0a47710964f with delete_type TABLET_DATA_DELETED (replaced by tablet 2083db0432024c898f63e5dbff47a2d9) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.186779 1159 raft_consensus.cc:357] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.186909 1159 raft_consensus.cc:383] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.186951 1159 raft_consensus.cc:738] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.187013 742 tablet_service.cc:1515] Processing DeleteTablet for tablet d331a13d13f34456be057dd9cc9fc23b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c51442a6a48ee8af1cbdcdb2b963c) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.187155 748 tablet_service.cc:1515] Processing DeleteTablet for tablet feeaa50e7f224da69128ea916e4145fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.187106 1159 consensus_queue.cc:260] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.187294 743 tablet_service.cc:1515] Processing DeleteTablet for tablet d60a756dd2ed4a2e96aaf3f13e3a90de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.187433 747 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ce14bc48f24499ad1a02fbd53fbe08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.187610 1159 ts_tablet_manager.cc:1428] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250629 01:52:05.189972 896 tablet_service.cc:1515] Processing DeleteTablet for tablet d4b4e75281ba4cd8a8e1f0a47710964f with delete_type TABLET_DATA_DELETED (replaced by tablet 2083db0432024c898f63e5dbff47a2d9) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.190181 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 5ab0aa4e74d345b2a214b90b5018c82b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:50 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.190344 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 689adaf0c7564d9ba0b31061e4e2d8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.190493 896 tablet_service.cc:1515] Processing DeleteTablet for tablet 50d23f589de84dd5b150a17cee00c411 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.190659 896 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ce14bc48f24499ad1a02fbd53fbe08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.191262 884 tablet_service.cc:1515] Processing DeleteTablet for tablet d331a13d13f34456be057dd9cc9fc23b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c51442a6a48ee8af1cbdcdb2b963c) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.191373 879 tablet_service.cc:1515] Processing DeleteTablet for tablet feeaa50e7f224da69128ea916e4145fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.191462 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 07e0727102d54294a7c13a4aeef311e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.191545 880 tablet_service.cc:1515] Processing DeleteTablet for tablet ecbfc4e45dc9435bbd7a55ba7b7e4155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.191632 885 tablet_service.cc:1515] Processing DeleteTablet for tablet d60a756dd2ed4a2e96aaf3f13e3a90de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.198060 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d8190980352a4f78bfdc29a9f0e0c1cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.198908 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_594a5e0768744cc3bdfb59d5e908447f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.199265 2896 raft_consensus.cc:491] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.199368 2896 raft_consensus.cc:513] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.199851 2896 leader_election.cc:290] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.200268 916 raft_consensus.cc:3058] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.200338 1159 tablet_bootstrap.cc:492] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:05.201640 1159 tablet_bootstrap.cc:654] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.205737 2970 raft_consensus.cc:491] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.205883 2970 raft_consensus.cc:513] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.206477 2970 leader_election.cc:290] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.207134 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:05.207572 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33aa0a66cdbf49c5820e2b43b021f28b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.208851 712 leader_election.cc:343] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:05.210549 712 consensus_peers.cc:489] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:05.213670 2859 ts_tablet_manager.cc:1918] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.213768 2859 log.cc:1199] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/1fe0cd4b8f7d47268a80f645bc8b2b9d
I20250629 01:52:05.214066 2859 ts_tablet_manager.cc:1939] T 1fe0cd4b8f7d47268a80f645bc8b2b9d P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.214325 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1400) took 2063 ms. Trace:
I20250629 01:52:05.214393 2859 rpcz_store.cc:276] 0629 01:52:03.150848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.150946 (+ 98us) service_pool.cc:225] Handling call
0629 01:52:05.160721 (+2009775us) ts_tablet_manager.cc:1145] Deleting tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d
0629 01:52:05.160730 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.210853 (+ 50123us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.213584 (+ 2731us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.214309 (+ 725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2009644}
I20250629 01:52:05.214735 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 1fe0cd4b8f7d47268a80f645bc8b2b9d (table default.Table_6b2a627b815e45c480dcaafe356a5d50 [id=d5875760fd97485b88f3c69340dc7b01]) successfully deleted
W20250629 01:52:05.217288 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8190980352a4f78bfdc29a9f0e0c1cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:05.218355 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_594a5e0768744cc3bdfb59d5e908447f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.225919 2971 raft_consensus.cc:491] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.226069 2971 raft_consensus.cc:513] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.226616 2971 leader_election.cc:290] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.234632 2859 tablet_replica.cc:331] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.234825 2859 raft_consensus.cc:2241] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.234948 2859 raft_consensus.cc:2270] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.235363 2859 ts_tablet_manager.cc:1905] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.236435 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb26725bd175429d805947c4eda6e40a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.236776 712 leader_election.cc:343] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.247040 451 catalog_manager.cc:5674] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:05.252715 885 tablet_service.cc:1515] Processing DeleteTablet for tablet c5d45854191247bf8172d0bbfd01b6db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:52 UTC) from {username='slave'} at 127.0.0.1:56376
W20250629 01:52:05.252749 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb3f46b763b24f93ace0d1327a6be240 consensus state has no leader
I20250629 01:52:05.253063 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f27590f1ad614dafa2cb4866df1b83d0" } modify_external_catalogs: true
I20250629 01:52:05.253216 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f27590f1ad614dafa2cb4866df1b83d0" } modify_external_catalogs: true
I20250629 01:52:05.254458 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_680c76cd6af445d69aed85a081aa709d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.254873 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_680c76cd6af445d69aed85a081aa709d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.261066 2859 ts_tablet_manager.cc:1918] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.261154 2859 log.cc:1199] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f3bdc8bda8a940be865679a05b90bd08
I20250629 01:52:05.261420 2859 ts_tablet_manager.cc:1939] T f3bdc8bda8a940be865679a05b90bd08 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.261664 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1401) took 2110 ms. Trace:
I20250629 01:52:05.261723 2859 rpcz_store.cc:276] 0629 01:52:03.150962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.151098 (+ 136us) service_pool.cc:225] Handling call
0629 01:52:05.234592 (+2083494us) ts_tablet_manager.cc:1145] Deleting tablet f3bdc8bda8a940be865679a05b90bd08
0629 01:52:05.234601 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.237805 (+ 3204us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.260992 (+ 23187us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.261640 (+ 648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083396}
I20250629 01:52:05.262048 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f3bdc8bda8a940be865679a05b90bd08 (table default.Table_44f5d1913b0e4eedaac117acfdf02c22 [id=b6e2a7a8ccc245899af18bfd45e0f740]) successfully deleted
I20250629 01:52:05.264503 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_7dde0c66922743c1b227bc48f1adbd55" } new_table_name: "default.Table_8f8f680f0c43414292737f47bc1b7d3f" modify_external_catalogs: true
I20250629 01:52:05.265872 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_d0563326136c457aabee62f8d46d730c" } new_table_name: "default.Table_9a4c0ac7950945afb26d07949bdab304" modify_external_catalogs: true
I20250629 01:52:05.267397 2859 tablet_replica.cc:331] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.267537 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_438c0cf58ab646d1ae5cc60f28dc57f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.267879 2859 raft_consensus.cc:2241] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:52:05.268036 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_438c0cf58ab646d1ae5cc60f28dc57f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.268215 2859 raft_consensus.cc:2270] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.268844 2859 ts_tablet_manager.cc:1905] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.270743 2896 raft_consensus.cc:491] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.275246 2896 raft_consensus.cc:513] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.275699 2896 leader_election.cc:290] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.275786 2896 raft_consensus.cc:491] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.275857 2896 raft_consensus.cc:513] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.276288 2896 leader_election.cc:290] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:05.277455 850 leader_election.cc:343] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 689adaf0c7564d9ba0b31061e4e2d8cf
W20250629 01:52:05.277832 850 leader_election.cc:343] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: e5d27d658892496a9b19b57949a8399a
W20250629 01:52:05.278543 850 leader_election.cc:343] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 689adaf0c7564d9ba0b31061e4e2d8cf
I20250629 01:52:05.278622 850 leader_election.cc:304] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.279162 2859 ts_tablet_manager.cc:1918] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.280786 450 master_service.cc:948] ReplaceTablet: received request to replace tablet d2bd46f64051455a8e7fb306b5d0c810 from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:05.284041 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 3d87639255ec46d98fcfb0f9638ee39a from {username='slave'} at 127.0.0.1:56238
I20250629 01:52:05.286304 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d89e16db77bc49d08a15a65bc7ab7c54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.286763 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d89e16db77bc49d08a15a65bc7ab7c54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.298054 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet a582d4f907f340179024c0377d3893e2 (table default.Table_e26efcfb78bd4f3588d3f1de2dff7f3a [id=060d68b1f054440ea9f6e7977adcb982]). Expected version 1 got 0
I20250629 01:52:05.299299 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet e119711a47f14100a9455a1afd84b6d1 (table default.Table_d5d4609ee3c344c19f141a737ae4e9b7 [id=5ba5e48e7fd145bf80961b241679641e]). Expected version 2 got 0
I20250629 01:52:05.306391 454 catalog_manager.cc:5582] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:05.306577 752 tablet_service.cc:1468] Processing CreateTablet for tablet 17dd260d784949638e144393622468c0 (DEFAULT_TABLE table=default.Table_e88d20b80a994893996a9549bc4dbbb4 [id=9d985aa2a1b84032ab7e22257e985ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.306924 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17dd260d784949638e144393622468c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.307036 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet a43f6594e4844af580c99ea598272aae (table default.Table_31825683c00b4327866114e089fb02d7 [id=c1135b07e9bd455cb21ae0061c94e017]). Expected version 3 got 0
I20250629 01:52:05.307343 454 catalog_manager.cc:5582] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } health_report { overall_health: UNKNOWN } } }
I20250629 01:52:05.307597 454 catalog_manager.cc:5674] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891) does not have the latest schema for tablet b1ebbcb2c26b4667b2405154a9d8fccb (table default.Table_a2726ab5a15a4cf8833868289c7ca5ee [id=70f6cbc3d10848f9af1a4c482c27eeb8]). Expected version 1 got 0
I20250629 01:52:05.311029 749 tablet_service.cc:1468] Processing CreateTablet for tablet 710845c58a484dc09c10bb71090bc97a (DEFAULT_TABLE table=default.Table_8e651a7d91094b0d9f5526c179c6205d [id=85f85289bd8f41b4a7edca948466e76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.311300 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710845c58a484dc09c10bb71090bc97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.312294 756 tablet_service.cc:1468] Processing CreateTablet for tablet 1dfd1cc35af5492180ffa20e87cd5666 (DEFAULT_TABLE table=default.Table_39508ad73847431d95213096c80444d5 [id=d475df280ce54172a0372e0b3fa79650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.312541 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dfd1cc35af5492180ffa20e87cd5666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.316357 738 tablet_service.cc:1468] Processing CreateTablet for tablet 70a8dee9ed804663879a97e90f64a0f9 (DEFAULT_TABLE table=default.Table_dd26153e098741ffbe626a7171016faa [id=1f5d5199cedb44e091bc6af5f8bb39ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.316638 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a8dee9ed804663879a97e90f64a0f9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:05.281071 850 leader_election.cc:343] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: e5d27d658892496a9b19b57949a8399a
I20250629 01:52:05.281474 3028 raft_consensus.cc:2747] T 689adaf0c7564d9ba0b31061e4e2d8cf P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.317003 850 leader_election.cc:304] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.317178 3028 raft_consensus.cc:2747] T e5d27d658892496a9b19b57949a8399a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.317669 885 tablet_service.cc:1468] Processing CreateTablet for tablet fb6153d881124e5088e832d484a06389 (DEFAULT_TABLE table=default.Table_7c24c1ccd9e64a1d9c9b9cc86ff93ca6 [id=df207bd83a4548449972a7a522744eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.317976 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb6153d881124e5088e832d484a06389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.318948 896 tablet_service.cc:1468] Processing CreateTablet for tablet 17dd260d784949638e144393622468c0 (DEFAULT_TABLE table=default.Table_e88d20b80a994893996a9549bc4dbbb4 [id=9d985aa2a1b84032ab7e22257e985ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.319209 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17dd260d784949638e144393622468c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.320297 751 tablet_service.cc:1468] Processing CreateTablet for tablet c93d747c651b418583e5fa924c2ac93c (DEFAULT_TABLE table=default.Table_9ca0322f970846c9ba95443e38ed67e0 [id=8764e69b9d8944d6b32c5af4100e914f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.320299 888 tablet_service.cc:1468] Processing CreateTablet for tablet 70a8dee9ed804663879a97e90f64a0f9 (DEFAULT_TABLE table=default.Table_dd26153e098741ffbe626a7171016faa [id=1f5d5199cedb44e091bc6af5f8bb39ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.320552 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c93d747c651b418583e5fa924c2ac93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.320560 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a8dee9ed804663879a97e90f64a0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.321262 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1dfd1cc35af5492180ffa20e87cd5666 (DEFAULT_TABLE table=default.Table_39508ad73847431d95213096c80444d5 [id=d475df280ce54172a0372e0b3fa79650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.321506 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dfd1cc35af5492180ffa20e87cd5666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.323961 881 tablet_service.cc:1468] Processing CreateTablet for tablet c93d747c651b418583e5fa924c2ac93c (DEFAULT_TABLE table=default.Table_9ca0322f970846c9ba95443e38ed67e0 [id=8764e69b9d8944d6b32c5af4100e914f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.324218 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c93d747c651b418583e5fa924c2ac93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.324293 744 tablet_service.cc:1468] Processing CreateTablet for tablet bda7ce41a76d4adc9313dfa6b0b08f93 (DEFAULT_TABLE table=default.Table_da26ce67e0ca4692bb783caaf5e530d7 [id=c4c1f090573840719298f6df23c7a1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.324532 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda7ce41a76d4adc9313dfa6b0b08f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.328281 878 tablet_service.cc:1468] Processing CreateTablet for tablet bda7ce41a76d4adc9313dfa6b0b08f93 (DEFAULT_TABLE table=default.Table_da26ce67e0ca4692bb783caaf5e530d7 [id=c4c1f090573840719298f6df23c7a1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.328301 757 tablet_service.cc:1468] Processing CreateTablet for tablet 364afe732d7c4570a2f8602d2326dc8c (DEFAULT_TABLE table=default.Table_d3f10c12a5284e1aa89799a41ea487dd [id=3765d4b0f8fc4b7b80631280012fa8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.328536 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda7ce41a76d4adc9313dfa6b0b08f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.328756 750 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae7af185c844143bea797721cdb691f (DEFAULT_TABLE table=default.Table_849da15438e74c09b66190ea520f89bb [id=132c4ee18aa24264ba3e0820df8d1fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.328837 749 tablet_service.cc:1468] Processing CreateTablet for tablet a4d34f52d5464a1db746604c98676afc (DEFAULT_TABLE table=default.Table_292a95f97cbd424ea5aba15748ff9ee4 [id=13510a9403fa49e297db18d8d423fdcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.329155 753 tablet_service.cc:1468] Processing CreateTablet for tablet b4a188cc5f0340d2834cdccbee198099 (DEFAULT_TABLE table=default.Table_0f315136e0174620a55372819dd1c75b [id=4260a740d3b44ae5974d482b26ad109c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.329505 740 tablet_service.cc:1468] Processing CreateTablet for tablet b42151180af543aba4f7282708fa4199 (DEFAULT_TABLE table=default.Table_0710fe46a5274c88bdbbc51b5cc28aa9 [id=122a73c9447c4a65be1ef176676f6c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.330185 895 tablet_service.cc:1468] Processing CreateTablet for tablet ff1d136016f049749a05978a867bc3a3 (DEFAULT_TABLE table=default.Table_0d7a6dc928ef432996bb10611192b6a9 [id=7c468470486243029ca4d816d603f011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.330415 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1d136016f049749a05978a867bc3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.330837 746 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc94a84fa7e4f678af1909ae167a0b6 (DEFAULT_TABLE table=default.Table_96dae305586a4e209ec51d48ff898fe5 [id=a2ec30e439bd4f9089cdb68e1b4e4965]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.332129 886 tablet_service.cc:1468] Processing CreateTablet for tablet 364afe732d7c4570a2f8602d2326dc8c (DEFAULT_TABLE table=default.Table_d3f10c12a5284e1aa89799a41ea487dd [id=3765d4b0f8fc4b7b80631280012fa8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.332257 880 tablet_service.cc:1468] Processing CreateTablet for tablet 5b294e4a35a542fd9fa803ad10d1b1d0 (DEFAULT_TABLE table=default.Table_c950187cddb24b5eabf5f6d04516faaf [id=cb0a909d5450498ca92ef9aa89969dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.332363 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364afe732d7c4570a2f8602d2326dc8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.332475 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b294e4a35a542fd9fa803ad10d1b1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.332998 755 tablet_service.cc:1468] Processing CreateTablet for tablet dbca03a75c434223a686749656d7a6cf (DEFAULT_TABLE table=default.Table_0dac5dbb476a45639f013c63b022f815 [id=28081ba8bd49496b82f3bec8c2af25ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.336259 879 tablet_service.cc:1468] Processing CreateTablet for tablet 5ea1ec634e924fdc8d9fe91098639986 (DEFAULT_TABLE table=default.Table_63515e00ce034ce3af0c189f3ec838fd [id=4bb98b67585c4d21a1caee645e124faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.336493 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea1ec634e924fdc8d9fe91098639986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.340317 747 tablet_service.cc:1468] Processing CreateTablet for tablet fb6153d881124e5088e832d484a06389 (DEFAULT_TABLE table=default.Table_7c24c1ccd9e64a1d9c9b9cc86ff93ca6 [id=df207bd83a4548449972a7a522744eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.340720 884 tablet_service.cc:1468] Processing CreateTablet for tablet 206d6ac40a8846f4b935b6902f8aa52e (DEFAULT_TABLE table=default.Table_df7ca1b23ac246b2968ae5b317a47fb8 [id=7fc1079c4cb9440bb0657bffc2aadea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.340955 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 206d6ac40a8846f4b935b6902f8aa52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.342031 754 tablet_service.cc:1468] Processing CreateTablet for tablet 68fa5a879914405f8e4ca4110b8d74d4 (DEFAULT_TABLE table=default.Table_f42144c735164b209ece20fb363c95dc [id=eeaacf7febf740c8901eb3400c98c2e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.342814 892 tablet_service.cc:1468] Processing CreateTablet for tablet b42151180af543aba4f7282708fa4199 (DEFAULT_TABLE table=default.Table_0710fe46a5274c88bdbbc51b5cc28aa9 [id=122a73c9447c4a65be1ef176676f6c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.343039 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42151180af543aba4f7282708fa4199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.344311 748 tablet_service.cc:1468] Processing CreateTablet for tablet ff1d136016f049749a05978a867bc3a3 (DEFAULT_TABLE table=default.Table_0d7a6dc928ef432996bb10611192b6a9 [id=7c468470486243029ca4d816d603f011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.344693 743 tablet_service.cc:1468] Processing CreateTablet for tablet 5b294e4a35a542fd9fa803ad10d1b1d0 (DEFAULT_TABLE table=default.Table_c950187cddb24b5eabf5f6d04516faaf [id=cb0a909d5450498ca92ef9aa89969dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.345072 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5ea1ec634e924fdc8d9fe91098639986 (DEFAULT_TABLE table=default.Table_63515e00ce034ce3af0c189f3ec838fd [id=4bb98b67585c4d21a1caee645e124faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.346128 741 tablet_service.cc:1468] Processing CreateTablet for tablet ca89dd8775334836a577838d56d52713 (DEFAULT_TABLE table=default.Table_e258174bcbf9402daa2cb5b1127139b0 [id=a7b14794f6b642c0ba9a5ceee13e6613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.346251 745 tablet_service.cc:1468] Processing CreateTablet for tablet 206d6ac40a8846f4b935b6902f8aa52e (DEFAULT_TABLE table=default.Table_df7ca1b23ac246b2968ae5b317a47fb8 [id=7fc1079c4cb9440bb0657bffc2aadea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.347334 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364afe732d7c4570a2f8602d2326dc8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.347615 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc94a84fa7e4f678af1909ae167a0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.350411 878 tablet_service.cc:1468] Processing CreateTablet for tablet b4a188cc5f0340d2834cdccbee198099 (DEFAULT_TABLE table=default.Table_0f315136e0174620a55372819dd1c75b [id=4260a740d3b44ae5974d482b26ad109c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.350653 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a188cc5f0340d2834cdccbee198099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.352025 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42151180af543aba4f7282708fa4199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.352257 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68fa5a879914405f8e4ca4110b8d74d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.352697 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a188cc5f0340d2834cdccbee198099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.355166 896 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae7af185c844143bea797721cdb691f (DEFAULT_TABLE table=default.Table_849da15438e74c09b66190ea520f89bb [id=132c4ee18aa24264ba3e0820df8d1fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.355409 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae7af185c844143bea797721cdb691f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.356185 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae7af185c844143bea797721cdb691f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.356335 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbca03a75c434223a686749656d7a6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.360181 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1d136016f049749a05978a867bc3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.360198 2859 log.cc:1199] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/f94d8fc1b6254ba69acadcaff3613d97
I20250629 01:52:05.360747 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca89dd8775334836a577838d56d52713. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.364210 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea1ec634e924fdc8d9fe91098639986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.364362 2859 ts_tablet_manager.cc:1939] T f94d8fc1b6254ba69acadcaff3613d97 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.364603 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1402) took 2213 ms. Trace:
I20250629 01:52:05.364670 2859 rpcz_store.cc:276] 0629 01:52:03.151032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.151216 (+ 184us) service_pool.cc:225] Handling call
0629 01:52:05.267359 (+2116143us) ts_tablet_manager.cc:1145] Deleting tablet f94d8fc1b6254ba69acadcaff3613d97
0629 01:52:05.267367 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.276762 (+ 9395us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.279106 (+ 2344us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.364592 (+ 85486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116035}
I20250629 01:52:05.364897 2859 tablet_replica.cc:331] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.367782 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d34f52d5464a1db746604c98676afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.368197 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b294e4a35a542fd9fa803ad10d1b1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.372191 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb6153d881124e5088e832d484a06389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.365032 2859 raft_consensus.cc:2241] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.367244 3028 raft_consensus.cc:491] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.372299 3028 raft_consensus.cc:513] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:05.372851 3028 leader_election.cc:290] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.376186 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 206d6ac40a8846f4b935b6902f8aa52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.379966 749 tablet_service.cc:1468] Processing CreateTablet for tablet e8eb082d5f6b4467b585750eebc9fac3 (DEFAULT_TABLE table=default.Table_0070d2f697c64e168b71411e800f6823 [id=e27e8dfb38fb4a93b6030a969e8bae5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.380313 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8eb082d5f6b4467b585750eebc9fac3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:05.384370 850 leader_election.cc:343] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: ecbfc4e45dc9435bbd7a55ba7b7e4155
I20250629 01:52:05.386909 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3403dd561a174744a10a2d1a8180f81f (DEFAULT_TABLE table=default.Table_70b23bddafa9412880fb0cb669f1c6d0 [id=d81bc537b23844ba99e4d8bce4a206a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.387234 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3403dd561a174744a10a2d1a8180f81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.388743 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet f94d8fc1b6254ba69acadcaff3613d97 (table default.Table_3fc2c8b5061f4a6e8f46cd439cf5c776 [id=92d55fc2cf6343d7aac796eb0ddb93f8]) successfully deleted
I20250629 01:52:05.391824 881 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc94a84fa7e4f678af1909ae167a0b6 (DEFAULT_TABLE table=default.Table_96dae305586a4e209ec51d48ff898fe5 [id=a2ec30e439bd4f9089cdb68e1b4e4965]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.392130 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc94a84fa7e4f678af1909ae167a0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.392189 885 tablet_service.cc:1468] Processing CreateTablet for tablet e8eb082d5f6b4467b585750eebc9fac3 (DEFAULT_TABLE table=default.Table_0070d2f697c64e168b71411e800f6823 [id=e27e8dfb38fb4a93b6030a969e8bae5a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:05.392319 850 leader_election.cc:343] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: ecbfc4e45dc9435bbd7a55ba7b7e4155
I20250629 01:52:05.392392 850 leader_election.cc:304] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.392444 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8eb082d5f6b4467b585750eebc9fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.395576 886 tablet_service.cc:1468] Processing CreateTablet for tablet ca89dd8775334836a577838d56d52713 (DEFAULT_TABLE table=default.Table_e258174bcbf9402daa2cb5b1127139b0 [id=a7b14794f6b642c0ba9a5ceee13e6613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.395834 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca89dd8775334836a577838d56d52713. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.396157 3028 raft_consensus.cc:2747] T ecbfc4e45dc9435bbd7a55ba7b7e4155 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.400269 883 tablet_service.cc:1468] Processing CreateTablet for tablet e95c075d4eb74dbaa3f58cfc62de7137 (DEFAULT_TABLE table=default.Table_30f3d1b749884208a35c3f0a442b075b [id=97d5ae0d2bd8416ba2a2336060a0e0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.400539 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95c075d4eb74dbaa3f58cfc62de7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.403097 885 tablet_service.cc:1468] Processing CreateTablet for tablet e686cf80e8a74452994b12d600aa26bc (DEFAULT_TABLE table=default.Table_444d56c9689c4f32b190cbde3ab315af [id=9ea080f869fb46d682f1215aaca90f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.403366 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e686cf80e8a74452994b12d600aa26bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.403891 891 tablet_service.cc:1468] Processing CreateTablet for tablet 3403dd561a174744a10a2d1a8180f81f (DEFAULT_TABLE table=default.Table_70b23bddafa9412880fb0cb669f1c6d0 [id=d81bc537b23844ba99e4d8bce4a206a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.404160 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3403dd561a174744a10a2d1a8180f81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.408874 888 tablet_service.cc:1468] Processing CreateTablet for tablet 68fa5a879914405f8e4ca4110b8d74d4 (DEFAULT_TABLE table=default.Table_f42144c735164b209ece20fb363c95dc [id=eeaacf7febf740c8901eb3400c98c2e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.409022 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_9b8a8be075394faa80922439e514297a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.409490 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b8a8be075394faa80922439e514297a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.410952 746 tablet_service.cc:1468] Processing CreateTablet for tablet e686cf80e8a74452994b12d600aa26bc (DEFAULT_TABLE table=default.Table_444d56c9689c4f32b190cbde3ab315af [id=9ea080f869fb46d682f1215aaca90f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.411239 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e686cf80e8a74452994b12d600aa26bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.412045 892 tablet_service.cc:1468] Processing CreateTablet for tablet dbca03a75c434223a686749656d7a6cf (DEFAULT_TABLE table=default.Table_0dac5dbb476a45639f013c63b022f815 [id=28081ba8bd49496b82f3bec8c2af25ca]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:05.412127 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69464ae3512b47c29c9022c78efeef39 consensus state has no leader
I20250629 01:52:05.413102 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4505da86fcd44db7926a25eef6b827f1" } modify_external_catalogs: true
I20250629 01:52:05.420260 878 tablet_service.cc:1468] Processing CreateTablet for tablet 710845c58a484dc09c10bb71090bc97a (DEFAULT_TABLE table=default.Table_8e651a7d91094b0d9f5526c179c6205d [id=85f85289bd8f41b4a7edca948466e76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.420966 896 tablet_service.cc:1468] Processing CreateTablet for tablet a4d34f52d5464a1db746604c98676afc (DEFAULT_TABLE table=default.Table_292a95f97cbd424ea5aba15748ff9ee4 [id=13510a9403fa49e297db18d8d423fdcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.415278 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 3d87639255ec46d98fcfb0f9638ee39a
I20250629 01:52:05.424516 446 catalog_manager.cc:6652] ReplaceTablet: tablet 3d87639255ec46d98fcfb0f9638ee39a deleted and replaced by tablet d6b08d33d3694c53a9a56cb418f7da0f
I20250629 01:52:05.424685 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d87639255ec46d98fcfb0f9638ee39a with delete_type TABLET_DATA_DELETED (replaced by tablet d6b08d33d3694c53a9a56cb418f7da0f) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.424938 753 tablet_service.cc:1468] Processing CreateTablet for tablet 322587ba5ba3407ea6b36cf619482db2 (DEFAULT_TABLE table=default.Table_20da2869379148dba4a28145a932483b [id=6870d6fe2592462289767512af4c37f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.424998 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d87639255ec46d98fcfb0f9638ee39a with delete_type TABLET_DATA_DELETED (replaced by tablet d6b08d33d3694c53a9a56cb418f7da0f) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.425240 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322587ba5ba3407ea6b36cf619482db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.436636 2970 consensus_queue.cc:1035] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:05.443104 750 tablet_service.cc:1468] Processing CreateTablet for tablet e95c075d4eb74dbaa3f58cfc62de7137 (DEFAULT_TABLE table=default.Table_30f3d1b749884208a35c3f0a442b075b [id=97d5ae0d2bd8416ba2a2336060a0e0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.443490 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95c075d4eb74dbaa3f58cfc62de7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.450091 880 tablet_service.cc:1468] Processing CreateTablet for tablet cbbbc6ca150646b7b5ca540cf916ab7c (DEFAULT_TABLE table=default.Table_983d9cfd606940e1815eadca20ef3ecc [id=5066ed49b9c945e389fc61b64b081c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.450091 886 tablet_service.cc:1468] Processing CreateTablet for tablet 0f01c968c3504e87a1450683a6e8f22d (DEFAULT_TABLE table=default.Table_c62354526d4b47b6bbd63c4d14128c5b [id=3c20900a46a842da840f1082f272fc73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.450870 879 tablet_service.cc:1468] Processing CreateTablet for tablet fe6b9d8ccef54280860edb5097886b04 (DEFAULT_TABLE table=default.Table_9a4a654bd6864178bfacf5ebd390d8d2 [id=2e9d60a143ab4a359ded167833630bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.451485 895 tablet_service.cc:1468] Processing CreateTablet for tablet 322587ba5ba3407ea6b36cf619482db2 (DEFAULT_TABLE table=default.Table_20da2869379148dba4a28145a932483b [id=6870d6fe2592462289767512af4c37f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.452150 881 tablet_service.cc:1468] Processing CreateTablet for tablet e3c7c6a7ed9246ee8ed9ac649b5ec4c5 (DEFAULT_TABLE table=default.Table_69903da7c96841cc8b8e2b2f9fb281d1 [id=116bc65ea7b041078b5e9fe63adcc639]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.452132 884 tablet_service.cc:1468] Processing CreateTablet for tablet bf9fddfa57c9414ba95ec28ebd349551 (DEFAULT_TABLE table=default.Table_e7206ec77eb44e729b77a7bda6dea373 [id=4525666163d64ba981dfff1c095c6ee2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:05.452708 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac24b3d316ef4450b4aecd531cfd801a consensus state has no leader
I20250629 01:52:05.372305 2859 raft_consensus.cc:2270] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.453619 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0f01c968c3504e87a1450683a6e8f22d (DEFAULT_TABLE table=default.Table_c62354526d4b47b6bbd63c4d14128c5b [id=3c20900a46a842da840f1082f272fc73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.453987 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f01c968c3504e87a1450683a6e8f22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.453953 883 tablet_service.cc:1468] Processing CreateTablet for tablet d081b4e3bb094674a9ff4faf5182e021 (DEFAULT_TABLE table=default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.453949 2859 ts_tablet_manager.cc:1905] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.457387 2970 raft_consensus.cc:491] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.457520 2970 raft_consensus.cc:513] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.458160 2970 leader_election.cc:290] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.458487 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b54228f0013424588a355ceb17d9784" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.458879 712 leader_election.cc:343] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.458884 2534 consensus_queue.cc:1035] T bccdcd9de08648bb925db6fe56f97d20 P ef5c5979bb9e475f983af7c3e9e0e019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:05.459332 757 tablet_service.cc:1468] Processing CreateTablet for tablet cbbbc6ca150646b7b5ca540cf916ab7c (DEFAULT_TABLE table=default.Table_983d9cfd606940e1815eadca20ef3ecc [id=5066ed49b9c945e389fc61b64b081c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.459650 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbbc6ca150646b7b5ca540cf916ab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.465339 2534 raft_consensus.cc:491] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.465445 2534 raft_consensus.cc:513] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.465862 2534 leader_election.cc:290] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.466903 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02bd46ceac1e4766ada4c4d2e95fda00" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.467365 712 leader_election.cc:343] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.467830 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a5c12815f8f24b89aa5d482cde53223c" } modify_external_catalogs: true
I20250629 01:52:05.468019 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_a5c12815f8f24b89aa5d482cde53223c" } modify_external_catalogs: true
I20250629 01:52:05.468601 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 1bf612fea5b74a0bb78700cc07c4d68d
I20250629 01:52:05.470402 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf612fea5b74a0bb78700cc07c4d68d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.471882 756 tablet_service.cc:1468] Processing CreateTablet for tablet d081b4e3bb094674a9ff4faf5182e021 (DEFAULT_TABLE table=default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.472290 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d081b4e3bb094674a9ff4faf5182e021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.472640 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf612fea5b74a0bb78700cc07c4d68d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.475896 885 tablet_service.cc:1468] Processing CreateTablet for tablet 3db2d761823340a79987e395e446dcfd (DEFAULT_TABLE table=default.Table_8c5216b202244cbc976dcdac103bee27 [id=379f130c9fe94caba8b3f1502e5d38c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.476573 743 tablet_service.cc:1468] Processing CreateTablet for tablet e3c7c6a7ed9246ee8ed9ac649b5ec4c5 (DEFAULT_TABLE table=default.Table_69903da7c96841cc8b8e2b2f9fb281d1 [id=116bc65ea7b041078b5e9fe63adcc639]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.477015 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710845c58a484dc09c10bb71090bc97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.477423 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db2d761823340a79987e395e446dcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.477563 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f01c968c3504e87a1450683a6e8f22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.479455 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbca03a75c434223a686749656d7a6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.481046 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbbc6ca150646b7b5ca540cf916ab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.482836 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68fa5a879914405f8e4ca4110b8d74d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.491246 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322587ba5ba3407ea6b36cf619482db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.491348 877 tablet_service.cc:1468] Processing CreateTablet for tablet 30001ab22ba540bc860209512428734b (DEFAULT_TABLE table=default.Table_da8ce3889e724c1291c4dc0a46f206f2 [id=b37b2d06118646c6b7e406d02a800e34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.491616 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30001ab22ba540bc860209512428734b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.492406 893 tablet_service.cc:1468] Processing CreateTablet for tablet cc21a5857296434fa8bc056bc4cf6f3d (DEFAULT_TABLE table=default.Table_35d900b579bb493d9bd7cc39c09da9be [id=54ff1ea691864139a782ece2c5259fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.492673 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc21a5857296434fa8bc056bc4cf6f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.495868 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe6b9d8ccef54280860edb5097886b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.500653 748 tablet_service.cc:1468] Processing CreateTablet for tablet fe6b9d8ccef54280860edb5097886b04 (DEFAULT_TABLE table=default.Table_9a4a654bd6864178bfacf5ebd390d8d2 [id=2e9d60a143ab4a359ded167833630bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.501221 747 tablet_service.cc:1468] Processing CreateTablet for tablet bf9fddfa57c9414ba95ec28ebd349551 (DEFAULT_TABLE table=default.Table_e7206ec77eb44e729b77a7bda6dea373 [id=4525666163d64ba981dfff1c095c6ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.503914 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d081b4e3bb094674a9ff4faf5182e021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.504206 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c7c6a7ed9246ee8ed9ac649b5ec4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.513357 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d34f52d5464a1db746604c98676afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.513445 893 tablet_service.cc:1468] Processing CreateTablet for tablet cf6b9c8c6e2a4a51b4d4eaa7df29020d (DEFAULT_TABLE table=default.Table_c5f2016e1a584e959460454379b34a60 [id=ce6a01b8e429495aa657aa72be71b20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.513813 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6b9c8c6e2a4a51b4d4eaa7df29020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.514199 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4a26e492905840b5afc36d1598498bd0 (DEFAULT_TABLE table=default.Table_76d78a1d798b4ec5877b83b054814938 [id=57fda36826ce472e8e7c7047a4ba37b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.514430 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a26e492905840b5afc36d1598498bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.519249 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf9fddfa57c9414ba95ec28ebd349551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.525518 895 tablet_service.cc:1468] Processing CreateTablet for tablet 71169e43835f48a69756e43b298befe2 (DEFAULT_TABLE table=default.Table_0af022a022f8409c8373bc998d7ed733 [id=bb6081560768419284db441bf6999286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.526054 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71169e43835f48a69756e43b298befe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.189185 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet d4b4e75281ba4cd8a8e1f0a47710964f with delete_type TABLET_DATA_DELETED (replaced by tablet 2083db0432024c898f63e5dbff47a2d9) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.189293 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet d331a13d13f34456be057dd9cc9fc23b with delete_type TABLET_DATA_DELETED (replaced by tablet 574c51442a6a48ee8af1cbdcdb2b963c) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.189378 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet feeaa50e7f224da69128ea916e4145fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.189474 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet d60a756dd2ed4a2e96aaf3f13e3a90de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.529328 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b24606e6f4decb30364fa4fafc147) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.529479 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c2591c65d53449c8801778606f1f05f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.529594 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.529714 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 021da4395e414413b9b11ebcfbffcd19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:48 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.529850 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet d7cbc2e6148846f9a58239b089e5a052 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.529943 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 0467b153057f491e81681fa399a2a1a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:51 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.530212 1031 tablet_service.cc:1468] Processing CreateTablet for tablet a4d34f52d5464a1db746604c98676afc (DEFAULT_TABLE table=default.Table_292a95f97cbd424ea5aba15748ff9ee4 [id=13510a9403fa49e297db18d8d423fdcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.530521 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d34f52d5464a1db746604c98676afc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.529062 1158 rpcz_store.cc:276] 0629 01:52:03.440041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.440074 (+ 33us) service_pool.cc:225] Handling call
0629 01:52:05.147002 (+1706928us) ts_tablet_manager.cc:1145] Deleting tablet a75d7986152f427d88ed34a443128bc6
0629 01:52:05.147010 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.150513 (+ 3503us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.162904 (+ 12391us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.168307 (+ 5403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1706799}
I20250629 01:52:05.531066 1158 tablet_replica.cc:331] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:05.189563 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ce14bc48f24499ad1a02fbd53fbe08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.202695 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c62678645ff54aa6b3809212f4d66ec2" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.202946 1154 tablet_bootstrap.cc:492] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913: No bootstrap required, opened a new log
I20250629 01:52:05.208547 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33aa0a66cdbf49c5820e2b43b021f28b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.235860 2898 raft_consensus.cc:491] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.236558 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb26725bd175429d805947c4eda6e40a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.283308 2894 raft_consensus.cc:491] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.303086 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 5b294e4a35a542fd9fa803ad10d1b1d0 (DEFAULT_TABLE table=default.Table_c950187cddb24b5eabf5f6d04516faaf [id=cb0a909d5450498ca92ef9aa89969dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303299 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 206d6ac40a8846f4b935b6902f8aa52e (DEFAULT_TABLE table=default.Table_df7ca1b23ac246b2968ae5b317a47fb8 [id=7fc1079c4cb9440bb0657bffc2aadea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303395 1013 tablet_service.cc:1468] Processing CreateTablet for tablet ff1d136016f049749a05978a867bc3a3 (DEFAULT_TABLE table=default.Table_0d7a6dc928ef432996bb10611192b6a9 [id=7c468470486243029ca4d816d603f011]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303596 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 17dd260d784949638e144393622468c0 (DEFAULT_TABLE table=default.Table_e88d20b80a994893996a9549bc4dbbb4 [id=9d985aa2a1b84032ab7e22257e985ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303678 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 5ea1ec634e924fdc8d9fe91098639986 (DEFAULT_TABLE table=default.Table_63515e00ce034ce3af0c189f3ec838fd [id=4bb98b67585c4d21a1caee645e124faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303858 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 1dfd1cc35af5492180ffa20e87cd5666 (DEFAULT_TABLE table=default.Table_39508ad73847431d95213096c80444d5 [id=d475df280ce54172a0372e0b3fa79650]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303915 1014 tablet_service.cc:1468] Processing CreateTablet for tablet 70a8dee9ed804663879a97e90f64a0f9 (DEFAULT_TABLE table=default.Table_dd26153e098741ffbe626a7171016faa [id=1f5d5199cedb44e091bc6af5f8bb39ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.303957 1024 tablet_service.cc:1468] Processing CreateTablet for tablet fb6153d881124e5088e832d484a06389 (DEFAULT_TABLE table=default.Table_7c24c1ccd9e64a1d9c9b9cc86ff93ca6 [id=df207bd83a4548449972a7a522744eb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.304188 1029 tablet_service.cc:1468] Processing CreateTablet for tablet c93d747c651b418583e5fa924c2ac93c (DEFAULT_TABLE table=default.Table_9ca0322f970846c9ba95443e38ed67e0 [id=8764e69b9d8944d6b32c5af4100e914f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.304284 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5ae7af185c844143bea797721cdb691f (DEFAULT_TABLE table=default.Table_849da15438e74c09b66190ea520f89bb [id=132c4ee18aa24264ba3e0820df8d1fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.304468 1027 tablet_service.cc:1468] Processing CreateTablet for tablet bda7ce41a76d4adc9313dfa6b0b08f93 (DEFAULT_TABLE table=default.Table_da26ce67e0ca4692bb783caaf5e530d7 [id=c4c1f090573840719298f6df23c7a1f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.304540 1025 tablet_service.cc:1468] Processing CreateTablet for tablet b4a188cc5f0340d2834cdccbee198099 (DEFAULT_TABLE table=default.Table_0f315136e0174620a55372819dd1c75b [id=4260a740d3b44ae5974d482b26ad109c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.307224 1015 tablet_service.cc:1468] Processing CreateTablet for tablet b42151180af543aba4f7282708fa4199 (DEFAULT_TABLE table=default.Table_0710fe46a5274c88bdbbc51b5cc28aa9 [id=122a73c9447c4a65be1ef176676f6c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.307483 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 710845c58a484dc09c10bb71090bc97a (DEFAULT_TABLE table=default.Table_8e651a7d91094b0d9f5526c179c6205d [id=85f85289bd8f41b4a7edca948466e76c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.307610 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 364afe732d7c4570a2f8602d2326dc8c (DEFAULT_TABLE table=default.Table_d3f10c12a5284e1aa89799a41ea487dd [id=3765d4b0f8fc4b7b80631280012fa8e4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:05.367645 988 consensus_peers.cc:489] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:05.407689 2760 raft_consensus.cc:491] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250629 01:52:05.429828 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:37891, user_credentials={real_user=slave}} blocked reactor thread for 43618.7us
I20250629 01:52:05.430230 3030 raft_consensus.cc:491] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.458686 1043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b54228f0013424588a355ceb17d9784" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.466132 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02bd46ceac1e4766ada4c4d2e95fda00" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.537997 850 leader_election.cc:304] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.538383 3028 raft_consensus.cc:2747] T c62678645ff54aa6b3809212f4d66ec2 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.539085 888 tablet_service.cc:1468] Processing CreateTablet for tablet 00719adff0ca447392211ee67b3b4119 (DEFAULT_TABLE table=default.Table_3d8b63e73ce543d2bbfa915ed1f776fb [id=55874f18582d4806b770e69d9fe68584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.539574 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00719adff0ca447392211ee67b3b4119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.543493 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_26700fd18fbf4cddb81d599cf016884c" } new_table_name: "default.Table_f5c0847f6df54671b9129b4cc64cfdd0" modify_external_catalogs: true
I20250629 01:52:05.544490 452 catalog_manager.cc:5674] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889) does not have the latest schema for tablet c84dfb60f67548ec9fa4c84c42e500c8 (table default.Table_5ae2ada98ea943a4bd45f7e5b21dd961 [id=764852c141de44e480459c9576cb46d2]). Expected version 1 got 0
I20250629 01:52:05.545590 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_8a96f5c56f134fc0a23f144e4f7f53f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.546021 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a96f5c56f134fc0a23f144e4f7f53f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.547840 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_82a5867209d44446a2e3db934706cdaa" } new_table_name: "default.Table_4a256a7939a94d2cafcaa8fda201977b" modify_external_catalogs: true
I20250629 01:52:05.548959 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet ac2270c5abd4456890378c7f82b5f663
I20250629 01:52:05.549885 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_c4735030f49f465e9678b5d9f2f281b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.551074 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4735030f49f465e9678b5d9f2f281b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.554323 883 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2270c5abd4456890378c7f82b5f663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.555176 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af7bb496140a4999b9999124452b40cd" } modify_external_catalogs: true
I20250629 01:52:05.555536 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af7bb496140a4999b9999124452b40cd" } modify_external_catalogs: true
I20250629 01:52:05.558467 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_095e7cfff96a42deb7f4800b2d974ee6" } new_table_name: "default.Table_656c0f3baa724164a4bfacb54da20465" modify_external_catalogs: true
I20250629 01:52:05.568349 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_5f267b12f94b4adcbd06ecf0540d6f89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.568711 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f267b12f94b4adcbd06ecf0540d6f89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.575230 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet d2bd46f64051455a8e7fb306b5d0c810
I20250629 01:52:05.575614 450 catalog_manager.cc:6652] ReplaceTablet: tablet d2bd46f64051455a8e7fb306b5d0c810 deleted and replaced by tablet 2d5898e5a95a4ce09e631e1c504c4083
I20250629 01:52:05.576057 878 tablet_service.cc:1515] Processing DeleteTablet for tablet d2bd46f64051455a8e7fb306b5d0c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d5898e5a95a4ce09e631e1c504c4083) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.531885 2758 raft_consensus.cc:491] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.576792 2758 raft_consensus.cc:513] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.531980 1154 ts_tablet_manager.cc:1397] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913: Time spent bootstrapping tablet: real 0.396s user 0.003s sys 0.000s
I20250629 01:52:05.532052 1048 raft_consensus.cc:2466] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:05.532125 2898 raft_consensus.cc:513] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.577718 2898 leader_election.cc:290] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.577816 2898 raft_consensus.cc:491] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.577831 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_4505da86fcd44db7926a25eef6b827f1" } modify_external_catalogs: true
I20250629 01:52:05.577896 2898 raft_consensus.cc:513] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.578347 2898 leader_election.cc:290] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.578436 2898 raft_consensus.cc:491] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.578509 2898 raft_consensus.cc:513] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.578868 2898 leader_election.cc:290] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.532210 2894 raft_consensus.cc:513] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.579180 1051 raft_consensus.cc:2466] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:05.579412 2894 leader_election.cc:290] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:05.533622 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 165984us
I20250629 01:52:05.566601 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 12b85f220f3947ca8110f2db937e7b68 with delete_type TABLET_DATA_DELETED (replaced by tablet 999dae3757ad4c57bf4e2a92ee5203ca) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.531559 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 7dc94a84fa7e4f678af1909ae167a0b6 (DEFAULT_TABLE table=default.Table_96dae305586a4e209ec51d48ff898fe5 [id=a2ec30e439bd4f9089cdb68e1b4e4965]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.532186 1044 raft_consensus.cc:2466] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:05.532385 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b294e4a35a542fd9fa803ad10d1b1d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.534418 3030 raft_consensus.cc:513] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.580884 1043 raft_consensus.cc:2466] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:05.580991 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70a8dee9ed804663879a97e90f64a0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.533679 2760 raft_consensus.cc:513] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.581370 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 710845c58a484dc09c10bb71090bc97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.581555 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff1d136016f049749a05978a867bc3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.581799 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42151180af543aba4f7282708fa4199. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.581924 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae7af185c844143bea797721cdb691f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.582093 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ea1ec634e924fdc8d9fe91098639986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.582232 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb6153d881124e5088e832d484a06389. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.582453 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dfd1cc35af5492180ffa20e87cd5666. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.582603 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17dd260d784949638e144393622468c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.582736 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bda7ce41a76d4adc9313dfa6b0b08f93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.582867 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4a188cc5f0340d2834cdccbee198099. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.583662 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 364afe732d7c4570a2f8602d2326dc8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.584338 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc94a84fa7e4f678af1909ae167a0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.587671 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_e6d309d83afc49a9aa168e8188a919f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.588032 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6d309d83afc49a9aa168e8188a919f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.588256 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 206d6ac40a8846f4b935b6902f8aa52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.531330 1158 raft_consensus.cc:2241] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.588797 1158 raft_consensus.cc:2270] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.589236 1158 ts_tablet_manager.cc:1905] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.589367 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c93d747c651b418583e5fa924c2ac93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.589666 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3db2d761823340a79987e395e446dcfd (DEFAULT_TABLE table=default.Table_8c5216b202244cbc976dcdac103bee27 [id=379f130c9fe94caba8b3f1502e5d38c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.590639 2548 raft_consensus.cc:491] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.590787 2548 raft_consensus.cc:513] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.591115 1159 tablet_bootstrap.cc:492] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.591234 1159 ts_tablet_manager.cc:1397] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.391s user 0.013s sys 0.013s
I20250629 01:52:05.591727 3028 raft_consensus.cc:491] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.591823 3028 raft_consensus.cc:513] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } }
I20250629 01:52:05.592283 3028 leader_election.cc:290] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.592778 1159 raft_consensus.cc:357] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.592959 1159 raft_consensus.cc:383] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.593036 1159 raft_consensus.cc:738] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.593240 1159 consensus_queue.cc:260] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.601361 1159 ts_tablet_manager.cc:1428] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250629 01:52:05.601678 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 541728ff8dc3471ba90784b4a769c8e1
I20250629 01:52:05.604928 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_4256166d620c4234b09855f4e46afd3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.605656 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_d49a666bc540427f9646aa2e103828b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.606026 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d49a666bc540427f9646aa2e103828b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.606843 896 tablet_service.cc:1515] Processing DeleteTablet for tablet dc841eb171d04b118b1e705ceed4c861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.607028 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 541728ff8dc3471ba90784b4a769c8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.607795 1023 tablet_service.cc:1468] Processing CreateTablet for tablet dbca03a75c434223a686749656d7a6cf (DEFAULT_TABLE table=default.Table_0dac5dbb476a45639f013c63b022f815 [id=28081ba8bd49496b82f3bec8c2af25ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.608176 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbca03a75c434223a686749656d7a6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.609375 2548 leader_election.cc:290] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.610108 1159 tablet_bootstrap.cc:492] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
W20250629 01:52:05.610469 712 consensus_peers.cc:489] T b1ebbcb2c26b4667b2405154a9d8fccb P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:05.611359 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557ecee0d6e84f77ad10bbdd27859ee3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.611604 1032 tablet_service.cc:1468] Processing CreateTablet for tablet ca89dd8775334836a577838d56d52713 (DEFAULT_TABLE table=default.Table_e258174bcbf9402daa2cb5b1127139b0 [id=a7b14794f6b642c0ba9a5ceee13e6613]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.611932 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca89dd8775334836a577838d56d52713. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.612885 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e8eb082d5f6b4467b585750eebc9fac3 (DEFAULT_TABLE table=default.Table_0070d2f697c64e168b71411e800f6823 [id=e27e8dfb38fb4a93b6030a969e8bae5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.613183 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8eb082d5f6b4467b585750eebc9fac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.613271 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c7c6a7ed9246ee8ed9ac649b5ec4c5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:05.613528 712 leader_election.cc:343] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.614815 742 tablet_service.cc:1468] Processing CreateTablet for tablet 00719adff0ca447392211ee67b3b4119 (DEFAULT_TABLE table=default.Table_3d8b63e73ce543d2bbfa915ed1f776fb [id=55874f18582d4806b770e69d9fe68584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.615170 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00719adff0ca447392211ee67b3b4119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.616470 3028 raft_consensus.cc:491] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.616564 3028 raft_consensus.cc:513] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.616978 3028 leader_election.cc:290] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.617277 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e119711a47f14100a9455a1afd84b6d1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.617586 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "021da4395e414413b9b11ebcfbffcd19" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.617871 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18058ac8424043388bf4569e33d1da75" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.618213 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02bd46ceac1e4766ada4c4d2e95fda00" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.618659 988 leader_election.cc:343] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:05.619232 988 leader_election.cc:343] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:05.619519 988 leader_election.cc:343] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:05.619798 988 leader_election.cc:343] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
W20250629 01:52:05.620509 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4256166d620c4234b09855f4e46afd3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:05.621627 850 leader_election.cc:343] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 5ab0aa4e74d345b2a214b90b5018c82b
I20250629 01:52:05.621836 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf9fddfa57c9414ba95ec28ebd349551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.624181 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe6b9d8ccef54280860edb5097886b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.624871 753 tablet_service.cc:1468] Processing CreateTablet for tablet 71169e43835f48a69756e43b298befe2 (DEFAULT_TABLE table=default.Table_0af022a022f8409c8373bc998d7ed733 [id=bb6081560768419284db441bf6999286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.625135 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71169e43835f48a69756e43b298befe2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:05.626766 850 leader_election.cc:343] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: c5d45854191247bf8172d0bbfd01b6db
I20250629 01:52:05.627081 750 tablet_service.cc:1468] Processing CreateTablet for tablet 4a26e492905840b5afc36d1598498bd0 (DEFAULT_TABLE table=default.Table_76d78a1d798b4ec5877b83b054814938 [id=57fda36826ce472e8e7c7047a4ba37b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.627736 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 33aa0a66cdbf49c5820e2b43b021f28b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:56 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.627792 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 02bd46ceac1e4766ada4c4d2e95fda00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.628293 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3ba5887dffc1460a89208447ecc6a198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:55 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.628360 740 tablet_service.cc:1515] Processing DeleteTablet for tablet c5e9646c3ad04ed580f1d3402010dd38 with delete_type TABLET_DATA_DELETED (replaced by tablet dc1b24606e6f4decb30364fa4fafc147) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.637204 744 tablet_service.cc:1515] Processing DeleteTablet for tablet eb26725bd175429d805947c4eda6e40a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.637427 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d2bd46f64051455a8e7fb306b5d0c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d5898e5a95a4ce09e631e1c504c4083) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.637533 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 541728ff8dc3471ba90784b4a769c8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.637813 740 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2270c5abd4456890378c7f82b5f663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.640230 1156 ts_tablet_manager.cc:1918] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.640350 1156 log.cc:1199] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/43b845c3e7174618bd5c394d7206a634
I20250629 01:52:05.640626 1156 ts_tablet_manager.cc:1939] T 43b845c3e7174618bd5c394d7206a634 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:05.641834 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db2d761823340a79987e395e446dcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.642539 2859 ts_tablet_manager.cc:1918] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.642669 2859 log.cc:1199] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/45ca747f03524076b4112e4db78614fe
I20250629 01:52:05.643020 2859 ts_tablet_manager.cc:1939] T 45ca747f03524076b4112e4db78614fe P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.643296 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1404) took 2492 ms. Trace:
I20250629 01:52:05.645905 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a26e492905840b5afc36d1598498bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.656241 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b54228f0013424588a355ceb17d9784 with delete_type TABLET_DATA_DELETED (replaced by tablet ac560295fddd44368e6f9bc543f7cbfa) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.656543 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e95c075d4eb74dbaa3f58cfc62de7137 (DEFAULT_TABLE table=default.Table_30f3d1b749884208a35c3f0a442b075b [id=97d5ae0d2bd8416ba2a2336060a0e0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.656847 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e95c075d4eb74dbaa3f58cfc62de7137. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.643397 2859 rpcz_store.cc:276] 0629 01:52:03.151199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.151337 (+ 138us) service_pool.cc:225] Handling call
0629 01:52:05.364867 (+2213530us) ts_tablet_manager.cc:1145] Deleting tablet 45ca747f03524076b4112e4db78614fe
0629 01:52:05.364873 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.563458 (+198585us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.642471 (+ 79013us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.643282 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213454}
I20250629 01:52:05.657565 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 45ca747f03524076b4112e4db78614fe (table default.Table_e39f46841d454041bb064e74d3c27bc0 [id=f7eba54966074342a3632852c544ff35]) successfully deleted
I20250629 01:52:05.660245 2859 tablet_replica.cc:331] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.660449 2859 raft_consensus.cc:2241] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.660424 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 68fa5a879914405f8e4ca4110b8d74d4 (DEFAULT_TABLE table=default.Table_f42144c735164b209ece20fb363c95dc [id=eeaacf7febf740c8901eb3400c98c2e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.660645 2859 raft_consensus.cc:2270] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.660679 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68fa5a879914405f8e4ca4110b8d74d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.661155 2859 ts_tablet_manager.cc:1905] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.672241 1021 tablet_service.cc:1468] Processing CreateTablet for tablet e686cf80e8a74452994b12d600aa26bc (DEFAULT_TABLE table=default.Table_444d56c9689c4f32b190cbde3ab315af [id=9ea080f869fb46d682f1215aaca90f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.645133 1159 tablet_bootstrap.cc:654] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.667529 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b54228f0013424588a355ceb17d9784" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.667765 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7cbc2e6148846f9a58239b089e5a052" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.668040 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0467b153057f491e81681fa399a2a1a9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.675652 988 leader_election.cc:343] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.676194 746 tablet_service.cc:1468] Processing CreateTablet for tablet cf6b9c8c6e2a4a51b4d4eaa7df29020d (DEFAULT_TABLE table=default.Table_c5f2016e1a584e959460454379b34a60 [id=ce6a01b8e429495aa657aa72be71b20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.676517 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6b9c8c6e2a4a51b4d4eaa7df29020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.678154 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet a75d7986152f427d88ed34a443128bc6 (table default.Table_0ed0327b20c3440b9f184793c8e1ee7d [id=c5569004f0ea4c808d805df1734decfc]) successfully deleted
I20250629 01:52:05.680534 710 leader_election.cc:304] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.680610 1159 tablet_bootstrap.cc:492] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.680709 1159 ts_tablet_manager.cc:1397] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.071s user 0.001s sys 0.001s
I20250629 01:52:05.680960 710 leader_election.cc:304] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.681345 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 2045012c80d049eaabf7dec1df28b087 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:00 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.681402 2970 raft_consensus.cc:2802] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:05.681699 2970 raft_consensus.cc:491] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:05.681792 2970 raft_consensus.cc:3058] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.682087 1159 raft_consensus.cc:357] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.682202 1159 raft_consensus.cc:383] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.682238 1159 raft_consensus.cc:738] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.682404 1159 consensus_queue.cc:260] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.685077 1159 ts_tablet_manager.cc:1428] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250629 01:52:05.685283 1159 tablet_bootstrap.cc:492] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:05.689056 2970 raft_consensus.cc:513] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.689646 2970 leader_election.cc:290] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.690187 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb26725bd175429d805947c4eda6e40a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:05.690589 712 leader_election.cc:343] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.692713 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 322587ba5ba3407ea6b36cf619482db2 (DEFAULT_TABLE table=default.Table_20da2869379148dba4a28145a932483b [id=6870d6fe2592462289767512af4c37f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.693819 2971 raft_consensus.cc:2802] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:05.693908 2971 raft_consensus.cc:491] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:05.693986 2971 raft_consensus.cc:3058] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.694506 1159 tablet_bootstrap.cc:654] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.700441 1159 tablet_bootstrap.cc:492] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.700583 1159 ts_tablet_manager.cc:1397] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20250629 01:52:05.700966 2971 raft_consensus.cc:513] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.701165 1159 raft_consensus.cc:357] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.701323 1159 raft_consensus.cc:383] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.701408 1159 raft_consensus.cc:738] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.701526 2971 leader_election.cc:290] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.701931 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33aa0a66cdbf49c5820e2b43b021f28b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:05.702463 1159 consensus_queue.cc:260] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:05.707269 712 leader_election.cc:343] T 33aa0a66cdbf49c5820e2b43b021f28b P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.713428 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1440) took 3520 ms. Trace:
I20250629 01:52:05.713512 1158 ts_tablet_manager.cc:1918] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250629 01:52:05.713513 1156 rpcz_store.cc:276] 0629 01:52:02.192899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.193383 (+ 484us) service_pool.cc:225] Handling call
0629 01:52:05.162595 (+2969212us) ts_tablet_manager.cc:1145] Deleting tablet 43b845c3e7174618bd5c394d7206a634
0629 01:52:05.162602 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.590197 (+427595us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.640171 (+ 49974us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.713408 (+ 73237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969012}
I20250629 01:52:05.713827 1156 tablet_replica.cc:331] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:05.714035 1156 raft_consensus.cc:2241] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.714157 1156 raft_consensus.cc:2270] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.714385 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 43b845c3e7174618bd5c394d7206a634 (table default.Table_887a08ef61a94fddb7cbe5e3d8f80ac9 [id=ad6049db2ac64d5c86e06c58531e2120]) successfully deleted
I20250629 01:52:05.716851 1159 ts_tablet_manager.cc:1428] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250629 01:52:05.717675 710 leader_election.cc:304] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.717985 710 leader_election.cc:304] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.718470 2548 raft_consensus.cc:2802] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:05.718544 2548 raft_consensus.cc:491] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:05.718611 2548 raft_consensus.cc:3058] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.720965 2969 raft_consensus.cc:2802] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:05.721036 2969 raft_consensus.cc:491] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:05.721092 2969 raft_consensus.cc:3058] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.723084 2969 raft_consensus.cc:513] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.723536 2969 leader_election.cc:290] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.723860 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02bd46ceac1e4766ada4c4d2e95fda00" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:05.724259 712 leader_election.cc:343] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.724726 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557ecee0d6e84f77ad10bbdd27859ee3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
W20250629 01:52:05.726227 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf676660c9b44ffd87e365287e6fc0a9 consensus state has no leader
W20250629 01:52:05.732561 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b17c4768b35443cf8c289a96f2cf6d13 consensus state has no leader
I20250629 01:52:05.732815 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_24a17c99960b47728320a5f6217ba936"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.733237 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24a17c99960b47728320a5f6217ba936 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.735059 2548 raft_consensus.cc:513] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.735519 2548 leader_election.cc:290] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.736243 1156 ts_tablet_manager.cc:1905] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.736433 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b54228f0013424588a355ceb17d9784" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:05.736775 712 leader_election.cc:343] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.738759 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_50bf2655c9a2407180bd0a55e5bcb1cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.740270 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50bf2655c9a2407180bd0a55e5bcb1cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.740501 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 0f01c968c3504e87a1450683a6e8f22d (DEFAULT_TABLE table=default.Table_c62354526d4b47b6bbd63c4d14128c5b [id=3c20900a46a842da840f1082f272fc73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.741080 745 tablet_service.cc:1468] Processing CreateTablet for tablet cc21a5857296434fa8bc056bc4cf6f3d (DEFAULT_TABLE table=default.Table_35d900b579bb493d9bd7cc39c09da9be [id=54ff1ea691864139a782ece2c5259fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.741492 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc21a5857296434fa8bc056bc4cf6f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.742846 757 tablet_service.cc:1468] Processing CreateTablet for tablet 30001ab22ba540bc860209512428734b (DEFAULT_TABLE table=default.Table_da8ce3889e724c1291c4dc0a46f206f2 [id=b37b2d06118646c6b7e406d02a800e34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.743676 757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30001ab22ba540bc860209512428734b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.745736 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 059b0999c6114f4681be7b97cd9eb071
I20250629 01:52:05.752122 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 059b0999c6114f4681be7b97cd9eb071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.756475 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 059b0999c6114f4681be7b97cd9eb071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.757606 2970 raft_consensus.cc:491] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.757750 2970 raft_consensus.cc:513] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.758394 2970 leader_election.cc:290] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.758390 1014 tablet_service.cc:1468] Processing CreateTablet for tablet e3c7c6a7ed9246ee8ed9ac649b5ec4c5 (DEFAULT_TABLE table=default.Table_69903da7c96841cc8b8e2b2f9fb281d1 [id=116bc65ea7b041078b5e9fe63adcc639]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.758692 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43f6594e4844af580c99ea598272aae" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.758791 1020 tablet_service.cc:1468] Processing CreateTablet for tablet cbbbc6ca150646b7b5ca540cf916ab7c (DEFAULT_TABLE table=default.Table_983d9cfd606940e1815eadca20ef3ecc [id=5066ed49b9c945e389fc61b64b081c81]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.759018 2758 leader_election.cc:290] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.759168 2760 leader_election.cc:290] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:05.759189 712 leader_election.cc:343] T a43f6594e4844af580c99ea598272aae P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.759294 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ec92eddfabf40918a6cabaff5fe113a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.759603 1018 tablet_service.cc:1468] Processing CreateTablet for tablet d081b4e3bb094674a9ff4faf5182e021 (DEFAULT_TABLE table=default.Table_5b1107f5f7624470a33b0defd6eedd44 [id=dfc541fca28746b8ba7e1f81a24c7fce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.764979 1159 tablet_bootstrap.cc:492] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:05.768453 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_e33e44d3bb7d4104866f13a3bd938c15" } new_table_name: "default.Table_8441bf92de084874909a8ab6fc4f04e8" modify_external_catalogs: true
I20250629 01:52:05.768867 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e686cf80e8a74452994b12d600aa26bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:05.768910 988 leader_election.cc:343] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.768940 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 3403dd561a174744a10a2d1a8180f81f (DEFAULT_TABLE table=default.Table_70b23bddafa9412880fb0cb669f1c6d0 [id=d81bc537b23844ba99e4d8bce4a206a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250629 01:52:05.769040 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 92953.2us
W20250629 01:52:05.769616 988 leader_election.cc:343] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.772243 3030 leader_election.cc:290] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.774828 3028 raft_consensus.cc:491] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.774935 3028 raft_consensus.cc:513] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.775377 3028 leader_election.cc:290] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
W20250629 01:52:05.776176 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c5d983b4ddf4e6b824c644618998c81 has no consensus state
W20250629 01:52:05.778841 850 leader_election.cc:343] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: 43b845c3e7174618bd5c394d7206a634
W20250629 01:52:05.780469 1155 thread.cc:640] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.000s sys 0.006s
W20250629 01:52:05.780570 1155 thread.cc:606] log-alloc [worker] (thread pool) Time spent starting thread: real 0.619s user 0.000s sys 0.006s
I20250629 01:52:05.783131 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_8a1e4394a537495b9bf3f9e4335f7936" } new_table_name: "default.Table_8faaf5093a0b43a580c1d5d66262b03a" modify_external_catalogs: true
I20250629 01:52:05.783432 1159 tablet_bootstrap.cc:654] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.785087 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 43b845c3e7174618bd5c394d7206a634 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.786592 891 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a8a9bc691e4b158276f34a51c0dc61 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:46 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.791328 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_67461ab444b64f80854e45ddd5e4bcb8" } modify_external_catalogs: true
I20250629 01:52:05.791460 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_67461ab444b64f80854e45ddd5e4bcb8" } modify_external_catalogs: true
I20250629 01:52:05.793102 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 3db2d761823340a79987e395e446dcfd (DEFAULT_TABLE table=default.Table_8c5216b202244cbc976dcdac103bee27 [id=379f130c9fe94caba8b3f1502e5d38c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.795289 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 00719adff0ca447392211ee67b3b4119 (DEFAULT_TABLE table=default.Table_3d8b63e73ce543d2bbfa915ed1f776fb [id=55874f18582d4806b770e69d9fe68584]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.796444 2859 ts_tablet_manager.cc:1918] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.796536 2859 log.cc:1199] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/9fe9674ffeb44ba6ab60de59d4c08670
I20250629 01:52:05.796805 2859 ts_tablet_manager.cc:1939] T 9fe9674ffeb44ba6ab60de59d4c08670 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.797694 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1403) took 2646 ms. Trace:
I20250629 01:52:05.797772 2859 rpcz_store.cc:276] 0629 01:52:03.151091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.151436 (+ 345us) service_pool.cc:225] Handling call
0629 01:52:05.660202 (+2508766us) ts_tablet_manager.cc:1145] Deleting tablet 9fe9674ffeb44ba6ab60de59d4c08670
0629 01:52:05.660212 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.710061 (+ 49849us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.796361 (+ 86300us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.797681 (+ 1320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2508669}
I20250629 01:52:05.798036 2859 tablet_replica.cc:331] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.798197 2859 raft_consensus.cc:2241] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.798316 2859 raft_consensus.cc:2270] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.798749 2859 ts_tablet_manager.cc:1905] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.799551 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet 6a9509620c1c4fcea8fff19dab333f7a
I20250629 01:52:05.800930 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a9509620c1c4fcea8fff19dab333f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.801338 1155 tablet_bootstrap.cc:492] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019: No bootstrap required, opened a new log
I20250629 01:52:05.801636 1155 ts_tablet_manager.cc:1397] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent bootstrapping tablet: real 0.643s user 0.000s sys 0.009s
I20250629 01:52:05.803361 1155 raft_consensus.cc:357] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.803889 1155 raft_consensus.cc:383] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.804152 1155 raft_consensus.cc:738] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5c5979bb9e475f983af7c3e9e0e019, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.804338 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 9fe9674ffeb44ba6ab60de59d4c08670 (table default.Table_49d0e0f59d084d388696b7b11d6d50d7 [id=a46be286dc244265a1f250ecd48a65e9]) successfully deleted
I20250629 01:52:05.805106 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c7c6a7ed9246ee8ed9ac649b5ec4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.805085 1021 tablet_service.cc:1468] Processing CreateTablet for tablet cc21a5857296434fa8bc056bc4cf6f3d (DEFAULT_TABLE table=default.Table_35d900b579bb493d9bd7cc39c09da9be [id=54ff1ea691864139a782ece2c5259fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.806828 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cf6b9c8c6e2a4a51b4d4eaa7df29020d (DEFAULT_TABLE table=default.Table_c5f2016e1a584e959460454379b34a60 [id=ce6a01b8e429495aa657aa72be71b20a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.808930 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a9509620c1c4fcea8fff19dab333f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.810952 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_758849ecfdd541798af2cb9fd4f15ef0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.811359 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_758849ecfdd541798af2cb9fd4f15ef0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.811627 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eb26725bd175429d805947c4eda6e40a" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:05.811812 1048 raft_consensus.cc:3058] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.815459 1048 raft_consensus.cc:2466] T eb26725bd175429d805947c4eda6e40a P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:05.815616 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 3d87639255ec46d98fcfb0f9638ee39a with delete_type TABLET_DATA_DELETED (replaced by tablet d6b08d33d3694c53a9a56cb418f7da0f) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.818106 1154 raft_consensus.cc:357] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.818253 1154 raft_consensus.cc:383] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.818331 1154 raft_consensus.cc:738] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.818485 1154 consensus_queue.cc:260] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.818953 1154 ts_tablet_manager.cc:1428] T 6b0ef1ba46ac487ab1502c3d116814e1 P 4712643b602e4f898752fcab6a1e5913: Time spent starting tablet: real 0.242s user 0.001s sys 0.000s
I20250629 01:52:05.819237 1154 tablet_bootstrap.cc:492] T aef865f51a6b4191b2ab453c15094cc1 P 4712643b602e4f898752fcab6a1e5913: Bootstrap starting.
I20250629 01:52:05.819967 1052 raft_consensus.cc:2466] T 557ecee0d6e84f77ad10bbdd27859ee3 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:05.820447 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet ac2270c5abd4456890378c7f82b5f663 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.820925 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet d2bd46f64051455a8e7fb306b5d0c810 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d5898e5a95a4ce09e631e1c504c4083) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.821098 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 541728ff8dc3471ba90784b4a769c8e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.821861 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 059b0999c6114f4681be7b97cd9eb071 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.826325 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5d856d8a80084e8bae95e734dff72636" } modify_external_catalogs: true
I20250629 01:52:05.826455 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_5d856d8a80084e8bae95e734dff72636" } modify_external_catalogs: true
I20250629 01:52:05.828380 1013 tablet_service.cc:1468] Processing CreateTablet for tablet 71169e43835f48a69756e43b298befe2 (DEFAULT_TABLE table=default.Table_0af022a022f8409c8373bc998d7ed733 [id=bb6081560768419284db441bf6999286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.828570 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 4a26e492905840b5afc36d1598498bd0 (DEFAULT_TABLE table=default.Table_76d78a1d798b4ec5877b83b054814938 [id=57fda36826ce472e8e7c7047a4ba37b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.829877 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a43f6594e4844af580c99ea598272aae" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913" is_pre_election: true
I20250629 01:52:05.830070 1052 raft_consensus.cc:2466] T a43f6594e4844af580c99ea598272aae P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 0.
I20250629 01:52:05.831717 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02bd46ceac1e4766ada4c4d2e95fda00" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:05.831868 1044 raft_consensus.cc:3058] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.834316 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_28488d30183c4b499b52a80005451f96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.834398 1044 raft_consensus.cc:2466] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
W20250629 01:52:05.834717 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28488d30183c4b499b52a80005451f96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.834905 1027 tablet_service.cc:1468] Processing CreateTablet for tablet bf9fddfa57c9414ba95ec28ebd349551 (DEFAULT_TABLE table=default.Table_e7206ec77eb44e729b77a7bda6dea373 [id=4525666163d64ba981dfff1c095c6ee2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.836023 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b54228f0013424588a355ceb17d9784" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:05.836211 1051 raft_consensus.cc:3058] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.838635 1051 raft_consensus.cc:2466] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:05.836004 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00719adff0ca447392211ee67b3b4119. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.838973 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc21a5857296434fa8bc056bc4cf6f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.835969 1158 log.cc:1199] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/c5d45854191247bf8172d0bbfd01b6db
I20250629 01:52:05.839401 1158 ts_tablet_manager.cc:1939] T c5d45854191247bf8172d0bbfd01b6db P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
I20250629 01:52:05.840056 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1468) took 2399 ms. Trace:
I20250629 01:52:05.840148 1158 rpcz_store.cc:276] 0629 01:52:03.440413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.440446 (+ 33us) service_pool.cc:225] Handling call
0629 01:52:05.531003 (+2090557us) ts_tablet_manager.cc:1145] Deleting tablet c5d45854191247bf8172d0bbfd01b6db
0629 01:52:05.531010 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.670356 (+139346us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.713450 (+ 43094us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.840040 (+126590us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2090431}
I20250629 01:52:05.840488 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1457) took 2404 ms. Trace:
I20250629 01:52:05.840552 1158 rpcz_store.cc:276] 0629 01:52:03.436293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.442455 (+ 6162us) service_pool.cc:225] Handling call
0629 01:52:05.840386 (+2397931us) ts_tablet_manager.cc:1145] Deleting tablet 10920d584d0c456d8a5eb8efb81da36f
0629 01:52:05.840394 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.840475 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2397756}
I20250629 01:52:05.840881 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1470) took 2313 ms. Trace:
I20250629 01:52:05.840968 1158 rpcz_store.cc:276] 0629 01:52:03.527722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.529365 (+ 1643us) service_pool.cc:225] Handling call
0629 01:52:05.840729 (+2311364us) ts_tablet_manager.cc:1145] Deleting tablet 9ea65305417e45fdb1a47768cc1ee054
0629 01:52:05.840736 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.840828 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2286174}
I20250629 01:52:05.841260 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1469) took 2316 ms. Trace:
I20250629 01:52:05.841339 1158 rpcz_store.cc:276] 0629 01:52:03.524389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.524826 (+ 437us) service_pool.cc:225] Handling call
0629 01:52:05.841137 (+2316311us) ts_tablet_manager.cc:1145] Deleting tablet 6f7051dc43b84d7bbf70b51f0b9bced8
0629 01:52:05.841144 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.841209 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2279573}
I20250629 01:52:05.841578 1158 tablet_replica.cc:331] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: stopping tablet replica
I20250629 01:52:05.841763 1158 raft_consensus.cc:2241] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.841910 1158 raft_consensus.cc:2270] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.843490 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71169e43835f48a69756e43b298befe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.835985 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 1bf612fea5b74a0bb78700cc07c4d68d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:04 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.843741 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "33aa0a66cdbf49c5820e2b43b021f28b" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4712643b602e4f898752fcab6a1e5913"
I20250629 01:52:05.843895 1045 raft_consensus.cc:3058] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.846160 1045 raft_consensus.cc:2466] T 33aa0a66cdbf49c5820e2b43b021f28b P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef5c5979bb9e475f983af7c3e9e0e019 in term 1.
I20250629 01:52:05.846410 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 30001ab22ba540bc860209512428734b (DEFAULT_TABLE table=default.Table_da8ce3889e724c1291c4dc0a46f206f2 [id=b37b2d06118646c6b7e406d02a800e34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.847865 1154 tablet_bootstrap.cc:654] T aef865f51a6b4191b2ab453c15094cc1 P 4712643b602e4f898752fcab6a1e5913: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.848477 2859 ts_tablet_manager.cc:1918] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.848680 2859 log.cc:1199] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/1b0782c7e652419096710563930f2072
I20250629 01:52:05.849056 2859 ts_tablet_manager.cc:1939] T 1b0782c7e652419096710563930f2072 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.849371 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1399) took 2698 ms. Trace:
I20250629 01:52:05.847996 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d081b4e3bb094674a9ff4faf5182e021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.849571 2859 rpcz_store.cc:276] 0629 01:52:03.150758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.151246 (+ 488us) service_pool.cc:225] Handling call
0629 01:52:05.798001 (+2646755us) ts_tablet_manager.cc:1145] Deleting tablet 1b0782c7e652419096710563930f2072
0629 01:52:05.798009 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.822127 (+ 24118us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.844770 (+ 22643us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.849358 (+ 4588us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":132,"tablet-delete.queue_time_us":2646586}
I20250629 01:52:05.850050 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1405) took 2691 ms. Trace:
I20250629 01:52:05.850179 2859 rpcz_store.cc:276] 0629 01:52:03.158098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.158282 (+ 184us) service_pool.cc:225] Handling call
0629 01:52:05.849938 (+2691656us) ts_tablet_manager.cc:1145] Deleting tablet 790a1347e57f4cea895132564eafae9c
0629 01:52:05.849947 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.850035 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2691504}
I20250629 01:52:05.850559 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1406) took 2653 ms. Trace:
I20250629 01:52:05.850680 2859 rpcz_store.cc:276] 0629 01:52:03.197040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.197085 (+ 45us) service_pool.cc:225] Handling call
0629 01:52:05.850474 (+2653389us) ts_tablet_manager.cc:1145] Deleting tablet c4530a43bdeb47d3ad29b17750be12d2
0629 01:52:05.850482 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.850547 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2653253}
I20250629 01:52:05.851050 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1413) took 2403 ms. Trace:
I20250629 01:52:05.851177 2859 rpcz_store.cc:276] 0629 01:52:03.447922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.447958 (+ 36us) service_pool.cc:225] Handling call
0629 01:52:05.850972 (+2403014us) ts_tablet_manager.cc:1145] Deleting tablet 10920d584d0c456d8a5eb8efb81da36f
0629 01:52:05.850979 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.851038 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2402884}
I20250629 01:52:05.851493 2859 tablet_replica.cc:331] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.851779 2859 raft_consensus.cc:2241] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.852072 2859 raft_consensus.cc:2270] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.852113 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 1b0782c7e652419096710563930f2072 (table default.Table_de5d37756c7649fc9bf3ae6d5adf922f [id=6d20a5b14c084bd9bfe73ba92c415da7]) successfully deleted
I20250629 01:52:05.852201 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3403dd561a174744a10a2d1a8180f81f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250629 01:52:05.852317 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 790a1347e57f4cea895132564eafae9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 790a1347e57f4cea895132564eafae9c already in progress: creating tablet
W20250629 01:52:05.852479 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet c4530a43bdeb47d3ad29b17750be12d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4530a43bdeb47d3ad29b17750be12d2 already in progress: creating tablet
W20250629 01:52:05.852641 442 catalog_manager.cc:4908] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): delete failed for tablet 10920d584d0c456d8a5eb8efb81da36f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10920d584d0c456d8a5eb8efb81da36f already in progress: creating tablet
I20250629 01:52:05.848070 1032 tablet_service.cc:1468] Processing CreateTablet for tablet fe6b9d8ccef54280860edb5097886b04 (DEFAULT_TABLE table=default.Table_9a4a654bd6864178bfacf5ebd390d8d2 [id=2e9d60a143ab4a359ded167833630bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250629 01:52:05.856534 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe6b9d8ccef54280860edb5097886b04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.860791 1158 ts_tablet_manager.cc:1905] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:05.861521 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250629 01:52:05.861923 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250629 01:52:05.862040 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250629 01:52:05.863138 1159 tablet_bootstrap.cc:492] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.863242 1159 ts_tablet_manager.cc:1397] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.099s user 0.001s sys 0.005s
I20250629 01:52:05.863814 1159 raft_consensus.cc:357] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.863946 1159 raft_consensus.cc:383] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.864004 1159 raft_consensus.cc:738] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.864177 1159 consensus_queue.cc:260] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.865991 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf9fddfa57c9414ba95ec28ebd349551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.868043 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5e3cea3134a4dab8aa3e9b75293d5de" } modify_external_catalogs: true
I20250629 01:52:05.868184 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_b5e3cea3134a4dab8aa3e9b75293d5de" } modify_external_catalogs: true
I20250629 01:52:05.869262 1159 ts_tablet_manager.cc:1428] T 97d0fbb691514d51a1ca71f32a666c2d P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20250629 01:52:05.869546 1159 tablet_bootstrap.cc:492] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:05.870189 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30001ab22ba540bc860209512428734b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.871078 1159 tablet_bootstrap.cc:654] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.875010 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbbbc6ca150646b7b5ca540cf916ab7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.877265 2859 ts_tablet_manager.cc:1905] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.880183 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3db2d761823340a79987e395e446dcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.887537 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_45d3070d81804b73a194424a1cd32f8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:05.887957 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45d3070d81804b73a194424a1cd32f8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.888641 3028 raft_consensus.cc:491] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.888765 3028 raft_consensus.cc:513] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.889247 3028 leader_election.cc:290] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.890700 895 tablet_service.cc:1515] Processing DeleteTablet for tablet 97d0fbb691514d51a1ca71f32a666c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:49 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.892340 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f01c968c3504e87a1450683a6e8f22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.896867 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 322587ba5ba3407ea6b36cf619482db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.900458 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf6b9c8c6e2a4a51b4d4eaa7df29020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.901198 1159 tablet_bootstrap.cc:492] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.901347 1159 ts_tablet_manager.cc:1397] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.005s
I20250629 01:52:05.901983 1159 raft_consensus.cc:357] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.902170 1159 raft_consensus.cc:383] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.902592 1159 raft_consensus.cc:738] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.902863 1159 consensus_queue.cc:260] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.905221 440 catalog_manager.cc:4928] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): tablet c5d45854191247bf8172d0bbfd01b6db (table default.Table_4c419cdd9add4db7b041836405c4856b [id=7f7fa7fcbfc248319d2a0b8085312cdb]) successfully deleted
W20250629 01:52:05.905400 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 10920d584d0c456d8a5eb8efb81da36f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10920d584d0c456d8a5eb8efb81da36f already in progress: creating tablet
W20250629 01:52:05.905545 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 9ea65305417e45fdb1a47768cc1ee054 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ea65305417e45fdb1a47768cc1ee054 already in progress: creating tablet
W20250629 01:52:05.905670 440 catalog_manager.cc:4908] TS 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): delete failed for tablet 6f7051dc43b84d7bbf70b51f0b9bced8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f7051dc43b84d7bbf70b51f0b9bced8 already in progress: creating tablet
W20250629 01:52:05.906963 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28b171a57f5b4bfd95bb1324a5ee982c consensus state has no leader
I20250629 01:52:05.907455 1159 ts_tablet_manager.cc:1428] T d7e2c70221e94923b6a324a03f7133c6 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250629 01:52:05.907765 1159 tablet_bootstrap.cc:492] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:05.908998 1159 tablet_bootstrap.cc:654] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:05.914105 2859 ts_tablet_manager.cc:1918] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.914225 2859 log.cc:1199] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/327c303b6b9347f5b4cb78f74d41a44a
I20250629 01:52:05.914484 2859 ts_tablet_manager.cc:1939] T 327c303b6b9347f5b4cb78f74d41a44a P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.920318 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a26e492905840b5afc36d1598498bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250629 01:52:05.922344 1013 tablet_service.cc:1515] Processing DeleteTablet for tablet 6a9509620c1c4fcea8fff19dab333f7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.927300 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1414) took 2479 ms. Trace:
I20250629 01:52:05.929826 2859 rpcz_store.cc:276] 0629 01:52:03.448189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.448225 (+ 36us) service_pool.cc:225] Handling call
0629 01:52:05.851460 (+2403235us) ts_tablet_manager.cc:1145] Deleting tablet 327c303b6b9347f5b4cb78f74d41a44a
0629 01:52:05.851467 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.899982 (+ 48515us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.914039 (+ 14057us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.927287 (+ 13248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2403125}
I20250629 01:52:05.930238 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 327c303b6b9347f5b4cb78f74d41a44a (table default.Table_209b7bc3f1de4919a043f2ab3c1c5d6a [id=d62355a430724b96abb294c451c9bd58]) successfully deleted
I20250629 01:52:05.930522 2859 tablet_replica.cc:331] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.930469 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_aa4b088ab043449280d641b0ff03d845" } new_table_name: "default.Table_431323b5c7914e9983c85cbe56b5727a" modify_external_catalogs: true
I20250629 01:52:05.930759 2859 raft_consensus.cc:2241] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.930923 2859 raft_consensus.cc:2270] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.931986 2859 ts_tablet_manager.cc:1905] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.931569 1159 tablet_bootstrap.cc:492] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:05.932210 1159 ts_tablet_manager.cc:1397] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20250629 01:52:05.932798 1159 raft_consensus.cc:357] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.932994 1159 raft_consensus.cc:383] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:05.933082 1159 raft_consensus.cc:738] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:05.933125 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_f91234079da34b179a2115da66abaa37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250629 01:52:05.933313 1159 consensus_queue.cc:260] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:05.933879 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f91234079da34b179a2115da66abaa37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250629 01:52:05.938819 1159 ts_tablet_manager.cc:1428] T ec0ca04e5f6b42df8567deb53a3e5f53 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250629 01:52:05.939095 1159 tablet_bootstrap.cc:492] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:05.940670 1159 tablet_bootstrap.cc:654] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
W20250629 01:52:05.941040 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc1b24606e6f4decb30364fa4fafc147 consensus state has no leader
I20250629 01:52:05.949913 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_f802e382de6348e9b94cf6982926fa33" } new_table_name: "default.Table_6bb9d1e50ce741e183101852ee7d4e3e" modify_external_catalogs: true
I20250629 01:52:05.951438 895 tablet_service.cc:1515] Processing DeleteTablet for tablet ec0ca04e5f6b42df8567deb53a3e5f53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:51:47 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.957729 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet a46ad22ffb7a4bb9a74e1398e85ae3e5
I20250629 01:52:05.957953 2859 ts_tablet_manager.cc:1918] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.958038 2859 log.cc:1199] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/a014907d09c94745859436f1eb73b280
I20250629 01:52:05.961588 895 tablet_service.cc:1515] Processing DeleteTablet for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.958269 2859 ts_tablet_manager.cc:1939] T a014907d09c94745859436f1eb73b280 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.963644 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1415) took 2515 ms. Trace:
I20250629 01:52:05.963877 2859 rpcz_store.cc:276] 0629 01:52:03.448433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.448483 (+ 50us) service_pool.cc:225] Handling call
0629 01:52:05.930484 (+2482001us) ts_tablet_manager.cc:1145] Deleting tablet a014907d09c94745859436f1eb73b280
0629 01:52:05.930494 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.955495 (+ 25001us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.957887 (+ 2392us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.963629 (+ 5742us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2481890}
I20250629 01:52:05.964459 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet a014907d09c94745859436f1eb73b280 (table default.Table_98056b52f364497899c1998d23f58ce7 [id=d63189e5f0564d20a7f992ec881309fd]) successfully deleted
I20250629 01:52:05.964782 2859 tablet_replica.cc:331] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.965104 2859 raft_consensus.cc:2241] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.965654 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_29b731f4ff984ce5bb98b903fbb804c0" } modify_external_catalogs: true
I20250629 01:52:05.966068 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_29b731f4ff984ce5bb98b903fbb804c0" } modify_external_catalogs: true
I20250629 01:52:05.966557 2859 raft_consensus.cc:2270] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.966691 3030 consensus_queue.cc:1035] T c5771fdd618e45118df2b6dab7f668d9 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:05.967175 2859 ts_tablet_manager.cc:1905] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:05.967897 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:05.968914 1158 ts_tablet_manager.cc:1918] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:05.981436 2859 ts_tablet_manager.cc:1918] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250629 01:52:05.982321 2859 log.cc:1199] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/462789e8daef42a08ff8b2cd2ddd1c49
I20250629 01:52:05.982745 2859 ts_tablet_manager.cc:1939] T 462789e8daef42a08ff8b2cd2ddd1c49 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:05.983640 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1412) took 2536 ms. Trace:
I20250629 01:52:05.983902 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_af755a1cc02841f2a5ec1585c4d9d386" } new_table_name: "default.Table_a9903e15b2c7445db59b0927995aab13" modify_external_catalogs: true
I20250629 01:52:05.984334 2859 rpcz_store.cc:276] 0629 01:52:03.447614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.470951 (+ 23337us) service_pool.cc:225] Handling call
0629 01:52:05.964745 (+2493794us) ts_tablet_manager.cc:1145] Deleting tablet 462789e8daef42a08ff8b2cd2ddd1c49
0629 01:52:05.964753 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.973437 (+ 8684us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.981375 (+ 7938us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.983627 (+ 2252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2492841}
I20250629 01:52:05.984881 2859 tablet_replica.cc:331] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: stopping tablet replica
I20250629 01:52:05.985023 442 catalog_manager.cc:4928] TS 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): tablet 462789e8daef42a08ff8b2cd2ddd1c49 (table default.Table_7b3fdda8530a4fefbb38859d44f183af [id=d6d045dcc48741a1b48da383820fed8b]) successfully deleted
I20250629 01:52:05.985205 2859 raft_consensus.cc:2241] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250629 01:52:05.985319 2859 raft_consensus.cc:2270] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:05.985419 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a0b76160e22c46929936cb9682322fae: Sending DeleteTablet for 3 replicas of tablet aee0ea5e688c492885da7018e65d93e8
I20250629 01:52:05.892251 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a8a9bc691e4b158276f34a51c0dc61" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:05.804929 1155 consensus_queue.cc:260] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.985811 772 raft_consensus.cc:2466] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 0.
I20250629 01:52:05.986099 1155 ts_tablet_manager.cc:1428] T c2591c65d53449c8801778606f1f05f1 P ef5c5979bb9e475f983af7c3e9e0e019: Time spent starting tablet: real 0.184s user 0.000s sys 0.002s
I20250629 01:52:05.986164 850 leader_election.cc:304] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:05.986343 1155 tablet_bootstrap.cc:492] T e119711a47f14100a9455a1afd84b6d1 P ef5c5979bb9e475f983af7c3e9e0e019: Bootstrap starting.
I20250629 01:52:05.986410 3028 raft_consensus.cc:2802] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:05.986485 3028 raft_consensus.cc:491] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:05.986503 895 tablet_service.cc:1515] Processing DeleteTablet for tablet aee0ea5e688c492885da7018e65d93e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:56376
I20250629 01:52:05.986546 3028 raft_consensus.cc:3058] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.989176 3028 raft_consensus.cc:513] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.989730 3028 leader_election.cc:290] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:05.904129 710 leader_election.cc:304] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
W20250629 01:52:05.990959 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:33889, user_credentials={real_user=slave}} blocked reactor thread for 86872.4us
I20250629 01:52:05.991423 2548 raft_consensus.cc:2802] T eb26725bd175429d805947c4eda6e40a P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election won for term 1
W20250629 01:52:05.991541 850 leader_election.cc:343] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 5ab0aa4e74d345b2a214b90b5018c82b
I20250629 01:52:05.991583 710 leader_election.cc:304] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:05.991642 850 leader_election.cc:304] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.961344 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a46ad22ffb7a4bb9a74e1398e85ae3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.967592 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e119711a47f14100a9455a1afd84b6d1" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:05.992154 756 tablet_service.cc:1515] Processing DeleteTablet for tablet aee0ea5e688c492885da7018e65d93e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:38530
I20250629 01:52:05.992216 2969 raft_consensus.cc:491] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:05.992300 2969 raft_consensus.cc:513] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:05.992431 850 leader_election.cc:343] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: c5d45854191247bf8172d0bbfd01b6db
I20250629 01:52:05.992518 850 leader_election.cc:304] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:05.992713 2969 leader_election.cc:290] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.993175 2969 raft_consensus.cc:2802] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:05.993234 2969 raft_consensus.cc:491] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:05.993299 2969 raft_consensus.cc:3058] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.993844 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2045012c80d049eaabf7dec1df28b087" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
W20250629 01:52:05.994184 712 leader_election.cc:343] T 2045012c80d049eaabf7dec1df28b087 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.995800 2969 raft_consensus.cc:513] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:05.996989 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a8a9bc691e4b158276f34a51c0dc61" candidate_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:05.997135 772 raft_consensus.cc:3058] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:05.999506 772 raft_consensus.cc:2466] T d1a8a9bc691e4b158276f34a51c0dc61 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 443c4bd3c5d548ee8cd347d2b4de5293 in term 1.
I20250629 01:52:05.999994 3028 raft_consensus.cc:2747] T 5ab0aa4e74d345b2a214b90b5018c82b P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.000204 3028 raft_consensus.cc:2747] T c5d45854191247bf8172d0bbfd01b6db P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:05.967983 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "021da4395e414413b9b11ebcfbffcd19" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:06.000408 776 raft_consensus.cc:2466] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
I20250629 01:52:05.968335 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "18058ac8424043388bf4569e33d1da75" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:06.000618 2969 leader_election.cc:290] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Requested vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.968787 777 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0467b153057f491e81681fa399a2a1a9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:05.969558 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b54228f0013424588a355ceb17d9784" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:06.001024 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557ecee0d6e84f77ad10bbdd27859ee3" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:06.001113 773 raft_consensus.cc:2391] T 9b54228f0013424588a355ceb17d9784 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:52:05.838424 2970 raft_consensus.cc:491] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:06.001327 2970 raft_consensus.cc:513] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
W20250629 01:52:06.001389 712 leader_election.cc:343] T 557ecee0d6e84f77ad10bbdd27859ee3 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.001782 2970 leader_election.cc:290] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:05.968508 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "02bd46ceac1e4766ada4c4d2e95fda00" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:06.002038 771 raft_consensus.cc:2391] T 02bd46ceac1e4766ada4c4d2e95fda00 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4712643b602e4f898752fcab6a1e5913 in current term 1: Already voted for candidate ef5c5979bb9e475f983af7c3e9e0e019 in this term.
I20250629 01:52:06.002081 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0467b153057f491e81681fa399a2a1a9" candidate_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:05.969929 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7cbc2e6148846f9a58239b089e5a052" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
I20250629 01:52:06.002368 777 raft_consensus.cc:2466] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 0.
W20250629 01:52:06.002426 712 leader_election.cc:343] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:05.967787 1156 ts_tablet_manager.cc:1918] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:06.003516 1156 log.cc:1199] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-0/wal/wals/898b49812ece4b57a1a1015d9f26be2f
I20250629 01:52:06.004184 1156 ts_tablet_manager.cc:1939] T 898b49812ece4b57a1a1015d9f26be2f P ef5c5979bb9e475f983af7c3e9e0e019: Deleting consensus metadata
I20250629 01:52:06.004885 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1441) took 3811 ms. Trace:
I20250629 01:52:06.007284 3028 raft_consensus.cc:491] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:06.007438 3028 raft_consensus.cc:513] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.007943 3028 leader_election.cc:290] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889)
I20250629 01:52:06.011097 850 leader_election.cc:304] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
W20250629 01:52:06.011355 850 leader_election.cc:343] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d1a8a9bc691e4b158276f34a51c0dc61
W20250629 01:52:06.011636 850 leader_election.cc:343] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: 43b845c3e7174618bd5c394d7206a634
I20250629 01:52:06.011708 850 leader_election.cc:304] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.011909 3028 raft_consensus.cc:2802] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:06.011996 3052 raft_consensus.cc:2747] T 43b845c3e7174618bd5c394d7206a634 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.017702 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56238:
name: "default.Table_14bdb7211207485f810a2c6591c8f5c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250629 01:52:06.018167 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14bdb7211207485f810a2c6591c8f5c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250629 01:52:06.006625 712 consensus_peers.cc:489] T 056eeff228974a8e9036a37b58a2a675 P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:06.021817 850 leader_election.cc:343] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Not found: Tablet not found: dc841eb171d04b118b1e705ceed4c861
I20250629 01:52:06.021515 1156 rpcz_store.cc:276] 0629 01:52:02.193183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.193630 (+ 447us) service_pool.cc:225] Handling call
0629 01:52:05.713784 (+3520154us) ts_tablet_manager.cc:1145] Deleting tablet 898b49812ece4b57a1a1015d9f26be2f
0629 01:52:05.713793 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:05.738707 (+ 24914us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:05.967700 (+228993us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:06.004865 (+ 37165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3519992}
I20250629 01:52:06.022138 1156 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38530 (request call id 1433) took 3885 ms. Trace:
I20250629 01:52:06.022172 440 catalog_manager.cc:4928] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): tablet 898b49812ece4b57a1a1015d9f26be2f (table default.Table_57b8fd5efba54e2086b5e33f52312b34 [id=632421d87ddb4ca6bc86a29933d5a2bc]) successfully deleted
I20250629 01:52:06.022203 1156 rpcz_store.cc:276] 0629 01:52:02.136614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:02.193840 (+ 57226us) service_pool.cc:225] Handling call
0629 01:52:06.022042 (+3828202us) ts_tablet_manager.cc:1145] Deleting tablet a1da6e7dee904f23886f6d9d75c8e49f
0629 01:52:06.022050 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:06.022127 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3828029}
I20250629 01:52:06.022405 1156 tablet_replica.cc:331] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: stopping tablet replica
I20250629 01:52:06.022549 1156 raft_consensus.cc:2241] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250629 01:52:06.022541 440 catalog_manager.cc:4908] TS ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): delete failed for tablet a1da6e7dee904f23886f6d9d75c8e49f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6e7dee904f23886f6d9d75c8e49f already in progress: creating tablet
I20250629 01:52:06.022656 1156 raft_consensus.cc:2270] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250629 01:52:06.022171 2859 ts_tablet_manager.cc:1905] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250629 01:52:06.023067 1156 ts_tablet_manager.cc:1905] T 476a6cf28db04f67bc23ddfc5cacad87 P ef5c5979bb9e475f983af7c3e9e0e019: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250629 01:52:05.978559 988 consensus_peers.cc:489] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250629 01:52:06.024158 988 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:33393, user_credentials={real_user=slave}} blocked reactor thread for 45648.7us
I20250629 01:52:05.997637 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet aee0ea5e688c492885da7018e65d93e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-29 01:52:05 UTC) from {username='slave'} at 127.0.0.1:57444
I20250629 01:52:06.003034 3030 raft_consensus.cc:491] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250629 01:52:06.024335 3030 raft_consensus.cc:513] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.004195 2898 consensus_queue.cc:1035] T c5e9646c3ad04ed580f1d3402010dd38 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:06.024776 3030 leader_election.cc:290] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
W20250629 01:52:06.004731 986 leader_election.cc:343] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.005084 3035 consensus_queue.cc:1035] T 6ca02bcd9467496c9057b3de9c1b1725 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250629 01:52:06.024910 986 leader_election.cc:304] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.025142 3035 raft_consensus.cc:2747] T e119711a47f14100a9455a1afd84b6d1 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.025328 986 leader_election.cc:304] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:06.025439 1159 tablet_bootstrap.cc:492] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293: No bootstrap required, opened a new log
I20250629 01:52:06.025496 3035 raft_consensus.cc:2802] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:06.025555 3035 raft_consensus.cc:491] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:06.025606 3035 raft_consensus.cc:3058] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
W20250629 01:52:06.026342 986 leader_election.cc:343] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.026420 986 leader_election.cc:304] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.026621 3030 raft_consensus.cc:2747] T 18058ac8424043388bf4569e33d1da75 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.026680 986 leader_election.cc:304] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.026890 2760 raft_consensus.cc:2747] T 9b54228f0013424588a355ceb17d9784 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.026991 986 leader_election.cc:304] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.027206 2760 raft_consensus.cc:2747] T 02bd46ceac1e4766ada4c4d2e95fda00 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:06.027251 986 leader_election.cc:343] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.027318 986 leader_election.cc:304] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.027484 2760 raft_consensus.cc:2747] T d7cbc2e6148846f9a58239b089e5a052 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.027552 986 leader_election.cc:304] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters: 443c4bd3c5d548ee8cd347d2b4de5293
I20250629 01:52:06.027735 2760 raft_consensus.cc:2802] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250629 01:52:06.027694 3035 raft_consensus.cc:513] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.027789 2760 raft_consensus.cc:491] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250629 01:52:06.027837 2760 raft_consensus.cc:3058] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:06.028558 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c84dfb60f67548ec9fa4c84c42e500c8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" is_pre_election: true
W20250629 01:52:06.029800 986 leader_election.cc:343] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.029961 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "021da4395e414413b9b11ebcfbffcd19" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:06.030110 773 raft_consensus.cc:3058] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:06.026273 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c84dfb60f67548ec9fa4c84c42e500c8" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" is_pre_election: true
I20250629 01:52:06.031216 2760 raft_consensus.cc:513] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.031277 1159 ts_tablet_manager.cc:1397] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent bootstrapping tablet: real 0.092s user 0.006s sys 0.010s
I20250629 01:52:06.031670 2760 leader_election.cc:290] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Requested vote from peers ef5c5979bb9e475f983af7c3e9e0e019 (127.0.104.1:37891), 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393)
I20250629 01:52:06.031848 1159 raft_consensus.cc:357] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.031991 1159 raft_consensus.cc:383] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250629 01:52:06.032039 1159 raft_consensus.cc:738] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c4bd3c5d548ee8cd347d2b4de5293, State: Initialized, Role: FOLLOWER
I20250629 01:52:06.032215 773 raft_consensus.cc:2466] T 021da4395e414413b9b11ebcfbffcd19 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
W20250629 01:52:06.032436 988 leader_election.cc:343] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.032586 988 leader_election.cc:304] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913; no voters: 443c4bd3c5d548ee8cd347d2b4de5293, ef5c5979bb9e475f983af7c3e9e0e019
I20250629 01:52:06.032928 2760 raft_consensus.cc:2747] T c84dfb60f67548ec9fa4c84c42e500c8 P 4712643b602e4f898752fcab6a1e5913 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250629 01:52:06.033869 850 leader_election.cc:343] T d1a8a9bc691e4b158276f34a51c0dc61 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: d1a8a9bc691e4b158276f34a51c0dc61
I20250629 01:52:06.034019 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0467b153057f491e81681fa399a2a1a9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
W20250629 01:52:06.034116 850 leader_election.cc:343] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4712643b602e4f898752fcab6a1e5913 (127.0.104.3:33889): Not found: Tablet not found: dc841eb171d04b118b1e705ceed4c861
I20250629 01:52:06.034178 850 leader_election.cc:304] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 443c4bd3c5d548ee8cd347d2b4de5293; no voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019
W20250629 01:52:06.034580 988 leader_election.cc:343] T 0467b153057f491e81681fa399a2a1a9 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Illegal state: must be running to vote when last-logged opid is not known
I20250629 01:52:06.034790 3052 raft_consensus.cc:2747] T dc841eb171d04b118b1e705ceed4c861 P 443c4bd3c5d548ee8cd347d2b4de5293 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250629 01:52:06.034930 986 leader_election.cc:304] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4712643b602e4f898752fcab6a1e5913, ef5c5979bb9e475f983af7c3e9e0e019; no voters:
I20250629 01:52:06.035131 2898 raft_consensus.cc:2802] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 1 FOLLOWER]: Leader election won for term 1
I20250629 01:52:06.035279 2898 raft_consensus.cc:695] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [term 1 LEADER]: Becoming Leader. State: Replica: 4712643b602e4f898752fcab6a1e5913, State: Running, Role: LEADER
I20250629 01:52:06.035476 2898 consensus_queue.cc:237] T 021da4395e414413b9b11ebcfbffcd19 P 4712643b602e4f898752fcab6a1e5913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.036715 1158 log.cc:1199] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-2/wal/wals/f465d0bb46674763abfc8d93f45cd13d
I20250629 01:52:06.037037 1158 ts_tablet_manager.cc:1939] T f465d0bb46674763abfc8d93f45cd13d P 4712643b602e4f898752fcab6a1e5913: Deleting consensus metadata
W20250629 01:52:06.037497 712 consensus_peers.cc:489] T 8cced2e8c94345b88f388becb0060d7a P ef5c5979bb9e475f983af7c3e9e0e019 -> Peer 443c4bd3c5d548ee8cd347d2b4de5293 (127.0.104.2:33393): Couldn't send request to peer 443c4bd3c5d548ee8cd347d2b4de5293. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250629 01:52:06.037665 1158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57444 (request call id 1471) took 2455 ms. Trace:
I20250629 01:52:06.037915 1159 consensus_queue.cc:260] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef5c5979bb9e475f983af7c3e9e0e019" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37891 } } peers { permanent_uuid: "4712643b602e4f898752fcab6a1e5913" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33889 } } peers { permanent_uuid: "443c4bd3c5d548ee8cd347d2b4de5293" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33393 } }
I20250629 01:52:06.015164 1155 tablet_bootstrap.cc:654] T e119711a47f14100a9455a1afd84b6d1 P ef5c5979bb9e475f983af7c3e9e0e019: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:06.042300 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0467b153057f491e81681fa399a2a1a9" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5c5979bb9e475f983af7c3e9e0e019"
I20250629 01:52:06.042440 773 raft_consensus.cc:3058] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 0 FOLLOWER]: Advancing to term 1
I20250629 01:52:06.043608 1159 ts_tablet_manager.cc:1428] T 476a6cf28db04f67bc23ddfc5cacad87 P 443c4bd3c5d548ee8cd347d2b4de5293: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20250629 01:52:06.045264 773 raft_consensus.cc:2466] T 0467b153057f491e81681fa399a2a1a9 P ef5c5979bb9e475f983af7c3e9e0e019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4712643b602e4f898752fcab6a1e5913 in term 1.
I20250629 01:52:06.045694 1159 tablet_bootstrap.cc:492] T 3ba5887dffc1460a89208447ecc6a198 P 443c4bd3c5d548ee8cd347d2b4de5293: Bootstrap starting.
I20250629 01:52:06.046985 1159 tablet_bootstrap.cc:654] T 3ba5887dffc1460a89208447ecc6a198 P 443c4bd3c5d548ee8cd347d2b4de5293: Neither blocks nor log segments found. Creating new log.
I20250629 01:52:06.055702 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_520e60b731224e1b98a5165831c7148f" } modify_external_catalogs: true
I20250629 01:52:06.055877 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56238:
table { table_name: "default.Table_520e60b731224e1b98a5165831c7148f" } modify_external_catalogs: true
I20250629 01:52:06.051367 916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "021da4395e414413b9b11ebcfbffcd19" candidate_uuid: "4712643b602e4f898752fcab6a1e5913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "443c4bd3c5d548ee8cd347d2b4de5293"
I20250629 01:52:06.056958 2859 ts_tablet_manager.cc:1918] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250629 01:52:06.057050 2859 log.cc:1199] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting WAL directory at /tmp/dist-test-task2ZKnTZ/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1751161895816882-416-0/minicluster-data/ts-1/wal/wals/e4d6f1eb43b24fda94aa0afcf4501195
I20250629 01:52:06.057307 2859 ts_tablet_manager.cc:1939] T e4d6f1eb43b24fda94aa0afcf4501195 P 443c4bd3c5d548ee8cd347d2b4de5293: Deleting consensus metadata
I20250629 01:52:06.057871 2859 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56376 (request call id 1416) took 2579 ms. Trace:
I20250629 01:52:06.057932 2859 rpcz_store.cc:276] 0629 01:52:03.478473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0629 01:52:03.480483 (+ 2010us) service_pool.cc:225] Handling call
0629 01:52:05.984844 (+2504361us) ts_tablet_manager.cc:1145] Deleting tablet e4d6f1eb43b24fda94aa0afcf4501195
0629 01:52:05.984852 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0629 01:52:06.027687 (+ 42835us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:06.056875 (+ 29188us) tablet_metadata.cc:641] Metadata flushed
0629 01:52:06.057856 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us"